[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:59:16 dist-test-slave-1dms krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:59:16 dist-test-slave-1dms krb5kdc[395](info): set up 2 sockets
Sep 29 20:59:16 dist-test-slave-1dms krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 20:59:17.694401 388 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 20:59:17 dist-test-slave-1dms krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:59:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:59:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:59:17.858330 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43915
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:59:18.173529 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:18.173904 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:18.174068 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:18.181164 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:59:18.181294 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:18.181401 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:18.181499 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:59:18.181595 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:59:18.191242 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:18.192576 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:18.193982 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:18.206979 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:18.207032 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:18.207063 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:18.207722 408 server_base.cc:1034] running on GCE node
I20260929 20:59:18.208491 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:18.209901 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:18.211098 408 hybrid_clock.cc:648] HybridClock initialized: now 1790715558210985 us; error 100 us; skew 500 ppm
I20260929 20:59:18.211431 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:59:18 dist-test-slave-1dms krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:59:18.218979 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:59:18.220371 408 webserver.cc:460] Webserver started at http://127.0.97.62:45381/ using document root <none> and password file <none>
I20260929 20:59:18.221098 408 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:18.221233 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:18.221519 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:18.224552 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "a67bf6627a1f48fcaf414a1447b26999"
format_stamp: "Formatted at 2026-09-29 20:59:18 on dist-test-slave-1dms"
server_key: "c856bd36ff4ec294a48d8a3f2dab13eb"
server_key_iv: "46cba1aa44f7fbc46562ac2548078bf6"
server_key_version: "encryptionkey@0"
I20260929 20:59:18.225504 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "a67bf6627a1f48fcaf414a1447b26999"
format_stamp: "Formatted at 2026-09-29 20:59:18 on dist-test-slave-1dms"
server_key: "c856bd36ff4ec294a48d8a3f2dab13eb"
server_key_iv: "46cba1aa44f7fbc46562ac2548078bf6"
server_key_version: "encryptionkey@0"
I20260929 20:59:18.230935 408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:59:18.234112 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:18.235345 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:59:18.235509 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "a67bf6627a1f48fcaf414a1447b26999"
format_stamp: "Formatted at 2026-09-29 20:59:18 on dist-test-slave-1dms"
server_key: "c856bd36ff4ec294a48d8a3f2dab13eb"
server_key_iv: "46cba1aa44f7fbc46562ac2548078bf6"
server_key_version: "encryptionkey@0"
I20260929 20:59:18.235666 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:18.249842 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:18.253646 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:18.253940 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:18.272644 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43915
I20260929 20:59:18.272671 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43915 every 8 connection(s)
I20260929 20:59:18.274081 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:59:18.277668 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:18.282631 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 408
I20260929 20:59:18.282922 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 20:59:18.284706 388 external_mini_cluster.cc:1428] Setting key e27c971cd564e8be8ea7a015078139c1
I20260929 20:59:18.287230 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a67bf6627a1f48fcaf414a1447b26999: Bootstrap starting.
I20260929 20:59:18.291402 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a67bf6627a1f48fcaf414a1447b26999: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:18.292363 475 log.cc:824] T 00000000000000000000000000000000 P a67bf6627a1f48fcaf414a1447b26999: Log is configured to *not* fsync() on all Append() calls
I20260929 20:59:18.295275 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a67bf6627a1f48fcaf414a1447b26999: No bootstrap required, opened a new log
I20260929 20:59:18.303213 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P a67bf6627a1f48fcaf414a1447b26999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a67bf6627a1f48fcaf414a1447b26999" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43915 } }
I20260929 20:59:18.303591 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P a67bf6627a1f48fcaf414a1447b26999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:18.303715 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P a67bf6627a1f48fcaf414a1447b26999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a67bf6627a1f48fcaf414a1447b26999, State: Initialized, Role: FOLLOWER
I20260929 20:59:18.304370 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P a67bf6627a1f48fcaf414a1447b26999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a67bf6627a1f48fcaf414a1447b26999" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43915 } }
I20260929 20:59:18.304636 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P a67bf6627a1f48fcaf414a1447b26999 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:59:18.304768 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P a67bf6627a1f48fcaf414a1447b26999 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:59:18.304971 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a67bf6627a1f48fcaf414a1447b26999 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:18.307098 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P a67bf6627a1f48fcaf414a1447b26999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a67bf6627a1f48fcaf414a1447b26999" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43915 } }
I20260929 20:59:18.307601 475 leader_election.cc:302] T 00000000000000000000000000000000 P a67bf6627a1f48fcaf414a1447b26999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a67bf6627a1f48fcaf414a1447b26999; no voters:
I20260929 20:59:18.308233 475 leader_election.cc:288] T 00000000000000000000000000000000 P a67bf6627a1f48fcaf414a1447b26999 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:59:18.308969 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a67bf6627a1f48fcaf414a1447b26999 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:59:18.310626 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P a67bf6627a1f48fcaf414a1447b26999 [term 1 LEADER]: Becoming Leader. State: Replica: a67bf6627a1f48fcaf414a1447b26999, State: Running, Role: LEADER
I20260929 20:59:18.310773 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P a67bf6627a1f48fcaf414a1447b26999 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:59:18.311156 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P a67bf6627a1f48fcaf414a1447b26999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a67bf6627a1f48fcaf414a1447b26999" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43915 } }
I20260929 20:59:18.313946 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P a67bf6627a1f48fcaf414a1447b26999 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a67bf6627a1f48fcaf414a1447b26999. Latest consensus state: current_term: 1 leader_uuid: "a67bf6627a1f48fcaf414a1447b26999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a67bf6627a1f48fcaf414a1447b26999" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43915 } } }
I20260929 20:59:18.314317 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P a67bf6627a1f48fcaf414a1447b26999 [sys.catalog]: This master's current role is: LEADER
I20260929 20:59:18.314776 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P a67bf6627a1f48fcaf414a1447b26999 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a67bf6627a1f48fcaf414a1447b26999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a67bf6627a1f48fcaf414a1447b26999" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43915 } } }
I20260929 20:59:18.315470 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P a67bf6627a1f48fcaf414a1447b26999 [sys.catalog]: This master's current role is: LEADER
I20260929 20:59:18.317926 489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 29 20:59:18 dist-test-slave-1dms krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:59:18.323500 489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:59:18.333350 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:18.288354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33182 (local address 127.0.97.62:43915)
0929 20:59:18.289652 (+ 1298us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:18.289668 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:18.290662 (+ 994us) server_negotiation.cc:407] Connection header received
0929 20:59:18.302782 (+ 12120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:18.302805 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:18.303184 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:18.303595 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:18.307349 (+ 3754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:18.310176 (+ 2827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:18.312916 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:18.314132 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:18.314347 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:18.323072 (+ 8725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:18.323081 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:18.323096 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:18.323143 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:18.326268 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:18.327452 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:18.327469 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:18.327480 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:18.327714 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:18.328384 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:18.328392 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:18.328397 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:18.329319 (+ 922us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:18.329702 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:18.331147 (+ 1445us) server_negotiation.cc:299] Negotiation successful
0929 20:59:18.331699 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1050,"thread_start_us":849,"threads_started":1}
I20260929 20:59:18.343119 489 catalog_manager.cc:1347] Generated new cluster ID: 462f9209707c42a9a6cea7f198870120
I20260929 20:59:18.343221 489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:59:18.390967 489 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:59:18.392920 489 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:59:18.401430 489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a67bf6627a1f48fcaf414a1447b26999: Generated new TSK 0
I20260929 20:59:18.402406 489 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:59:18.452522 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43915
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:59:18.780988 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:18.781308 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:18.781493 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:18.789146 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:18.789296 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:18.789481 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:59:18.799196 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43915
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:18.800496 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:18.801874 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:18.814754 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:18.814829 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:18.814776 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:18.815819 501 server_base.cc:1034] running on GCE node
I20260929 20:59:18.816370 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:18.817318 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:18.818501 501 hybrid_clock.cc:648] HybridClock initialized: now 1790715558818462 us; error 34 us; skew 500 ppm
I20260929 20:59:18.818799 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:59:18 dist-test-slave-1dms krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:59:18.821820 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:59:18.823604 501 webserver.cc:460] Webserver started at http://127.0.97.1:42701/ using document root <none> and password file <none>
I20260929 20:59:18.824286 501 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:18.824425 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:18.824725 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:18.827586 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-0/data/instance:
uuid: "490c8765c8684b818ff63a8e89940284"
format_stamp: "Formatted at 2026-09-29 20:59:18 on dist-test-slave-1dms"
server_key: "93d45f61ea1683db493cf8a76cbcdb84"
server_key_iv: "d5fef77fede416e12760488a9442b8f8"
server_key_version: "encryptionkey@0"
I20260929 20:59:18.828370 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-0/wal/instance:
uuid: "490c8765c8684b818ff63a8e89940284"
format_stamp: "Formatted at 2026-09-29 20:59:18 on dist-test-slave-1dms"
server_key: "93d45f61ea1683db493cf8a76cbcdb84"
server_key_iv: "d5fef77fede416e12760488a9442b8f8"
server_key_version: "encryptionkey@0"
I20260929 20:59:18.833454 501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 20:59:18.837036 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:18.838578 501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:59:18.838768 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-0/wal
uuid: "490c8765c8684b818ff63a8e89940284"
format_stamp: "Formatted at 2026-09-29 20:59:18 on dist-test-slave-1dms"
server_key: "93d45f61ea1683db493cf8a76cbcdb84"
server_key_iv: "d5fef77fede416e12760488a9442b8f8"
server_key_version: "encryptionkey@0"
I20260929 20:59:18.838980 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:18.859609 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:18.864349 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:18.864632 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:18.865624 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:59:18.866856 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:59:18.866956 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:18.867096 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:59:18.867162 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:18.901388 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45697
I20260929 20:59:18.901427 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45697 every 8 connection(s)
I20260929 20:59:18.902669 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:59:18.910746 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 501
I20260929 20:59:18.910959 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:59:18.911626 388 external_mini_cluster.cc:1428] Setting key b9fe754bc03ca9f16316d28d4696f1ae
Sep 29 20:59:18 dist-test-slave-1dms krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:59:18.923358 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:18.905221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55575 (local address 127.0.97.62:43915)
0929 20:59:18.905679 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:18.905684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:18.906678 (+ 994us) server_negotiation.cc:407] Connection header received
0929 20:59:18.908381 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:18.908385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:18.908444 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:18.908537 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:18.910076 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:18.911177 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:18.913578 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:18.914107 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:18.914249 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:18.917740 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:18.917746 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:18.917749 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:18.917780 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:18.920378 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:18.921166 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:18.921176 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:18.921179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:18.921285 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:18.921705 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:18.921710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:18.921713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:18.921964 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:18.922130 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:18.922786 (+ 656us) server_negotiation.cc:299] Negotiation successful
0929 20:59:18.922988 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":229,"threads_started":1}
I20260929 20:59:18.924276 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:18.905849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43915 (local address 127.0.97.1:55575)
0929 20:59:18.906509 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:18.906557 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:18.908149 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:18.908787 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:18.908796 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:18.909381 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:18.909928 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:18.909945 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:18.911317 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:18.911321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:18.913454 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:18.913467 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:18.914323 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:18.914329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:18.914515 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:18.915210 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:59:18.915237 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:18.917516 (+ 2279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:18.920522 (+ 3006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:18.920531 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:18.920556 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:18.921041 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:18.921391 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:18.921398 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:18.921401 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:18.921587 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:18.922109 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:18.922117 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:18.922431 (+ 314us) client_negotiation.cc:769] Sending connection context
0929 20:59:18.922692 (+ 261us) client_negotiation.cc:240] Negotiation successful
0929 20:59:18.923187 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":260,"threads_started":1}
I20260929 20:59:18.925871 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43915
I20260929 20:59:18.926172 629 heartbeater.cc:464] Registering TS with master...
I20260929 20:59:18.926847 629 heartbeater.cc:511] Master 127.0.97.62:43915 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 20:59:18.928943 440 ts_manager.cc:194] Registered new tserver with Master: 490c8765c8684b818ff63a8e89940284 (127.0.97.1:45697)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 20:59:18.930485 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55575
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:59:18.951426 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43915
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:59:19.267638 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:19.267910 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:19.268007 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:19.277796 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:19.277933 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:19.278096 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:59:19.290918 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43915
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:19.292610 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:19.294138 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:19.309957 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:19.310055 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:19.310081 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:19.310688 635 server_base.cc:1034] running on GCE node
I20260929 20:59:19.311301 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:19.312263 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:19.313448 635 hybrid_clock.cc:648] HybridClock initialized: now 1790715559313415 us; error 30 us; skew 500 ppm
I20260929 20:59:19.313791 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:59:19 dist-test-slave-1dms krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:59:19.316817 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:59:19.318315 635 webserver.cc:460] Webserver started at http://127.0.97.2:37367/ using document root <none> and password file <none>
I20260929 20:59:19.318988 635 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:19.319134 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:19.319432 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:19.322290 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-1/data/instance:
uuid: "b9d25a8f03c941d085915b148985db12"
format_stamp: "Formatted at 2026-09-29 20:59:19 on dist-test-slave-1dms"
server_key: "f9698bdf54ec8ecc307d380f8cd65169"
server_key_iv: "a9574fd7cdf3b051b72fc2b67206786f"
server_key_version: "encryptionkey@0"
I20260929 20:59:19.323187 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b9d25a8f03c941d085915b148985db12"
format_stamp: "Formatted at 2026-09-29 20:59:19 on dist-test-slave-1dms"
server_key: "f9698bdf54ec8ecc307d380f8cd65169"
server_key_iv: "a9574fd7cdf3b051b72fc2b67206786f"
server_key_version: "encryptionkey@0"
I20260929 20:59:19.328333 635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 20:59:19.331722 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:19.333084 635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 20:59:19.333251 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-1/wal
uuid: "b9d25a8f03c941d085915b148985db12"
format_stamp: "Formatted at 2026-09-29 20:59:19 on dist-test-slave-1dms"
server_key: "f9698bdf54ec8ecc307d380f8cd65169"
server_key_iv: "a9574fd7cdf3b051b72fc2b67206786f"
server_key_version: "encryptionkey@0"
I20260929 20:59:19.333447 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:19.351979 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:19.355957 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:19.356242 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:19.357192 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:59:19.358461 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:59:19.358559 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:19.358709 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:59:19.358780 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:19.393777 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38829
I20260929 20:59:19.393862 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38829 every 8 connection(s)
I20260929 20:59:19.395068 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:59:19.395869 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 635
I20260929 20:59:19.396073 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:59:19.396653 388 external_mini_cluster.cc:1428] Setting key d343a1f57ec6a4e61a571225a6fc7b43
Sep 29 20:59:19 dist-test-slave-1dms krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 20:59:19.417433 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:19.398161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50317 (local address 127.0.97.62:43915)
0929 20:59:19.398349 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:19.398355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:19.399721 (+ 1366us) server_negotiation.cc:407] Connection header received
0929 20:59:19.401584 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:19.401592 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:19.401704 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:19.401825 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:19.403243 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:19.403945 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:19.406510 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:19.406918 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:19.407015 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:19.411264 (+ 4249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:19.411272 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:19.411277 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:19.411314 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:19.414383 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:19.415093 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:19.415101 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:19.415109 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:19.415233 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:19.415673 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:19.415677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:19.415679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:19.415941 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:19.416094 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:19.416875 (+ 781us) server_negotiation.cc:299] Negotiation successful
0929 20:59:19.417097 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:59:19.418423 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:19.398693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43915 (local address 127.0.97.2:50317)
0929 20:59:19.399550 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:19.399595 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:19.401294 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:19.402031 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:19.402040 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:19.402519 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:19.403076 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:19.403092 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:19.404092 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:19.404101 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:19.406361 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:19.406378 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:19.407141 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:19.407150 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:19.407460 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:19.408179 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:59:19.408216 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:19.410921 (+ 2705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:19.414523 (+ 3602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:19.414533 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:19.414550 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:19.414963 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:19.415346 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:19.415350 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:19.415354 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:19.415540 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:19.416077 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:19.416085 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:19.416430 (+ 345us) client_negotiation.cc:769] Sending connection context
0929 20:59:19.416754 (+ 324us) client_negotiation.cc:240] Negotiation successful
0929 20:59:19.417298 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":379,"threads_started":1}
I20260929 20:59:19.419947 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43915
I20260929 20:59:19.420279 763 heartbeater.cc:464] Registering TS with master...
I20260929 20:59:19.421058 763 heartbeater.cc:511] Master 127.0.97.62:43915 requested a full tablet report, sending...
I20260929 20:59:19.422446 440 ts_manager.cc:194] Registered new tserver with Master: b9d25a8f03c941d085915b148985db12 (127.0.97.2:38829)
I20260929 20:59:19.423224 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50317
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:59:19.438271 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43915
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:59:19.761813 768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:19.762081 768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:19.762176 768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:19.769119 768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:19.769225 768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:19.769338 768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:59:19.778690 768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43915
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:19.779891 768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:19.781137 768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:19.794286 773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:19.794404 774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:19.794713 776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:19.795841 768 server_base.cc:1034] running on GCE node
I20260929 20:59:19.796427 768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:19.797437 768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:19.798672 768 hybrid_clock.cc:648] HybridClock initialized: now 1790715559798627 us; error 71 us; skew 500 ppm
I20260929 20:59:19.799067 768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:59:19 dist-test-slave-1dms krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:59:19.802280 768 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:59:19.803810 768 webserver.cc:460] Webserver started at http://127.0.97.3:46397/ using document root <none> and password file <none>
I20260929 20:59:19.804498 768 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:19.804636 768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:19.805007 768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:19.807772 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-2/data/instance:
uuid: "43bd71fcd46545adb6281ee6b06d839e"
format_stamp: "Formatted at 2026-09-29 20:59:19 on dist-test-slave-1dms"
server_key: "f645b2898f8b9025644ae7aebd024f5c"
server_key_iv: "54ed70a445ad4d76473e5a4bf322a322"
server_key_version: "encryptionkey@0"
I20260929 20:59:19.808526 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-2/wal/instance:
uuid: "43bd71fcd46545adb6281ee6b06d839e"
format_stamp: "Formatted at 2026-09-29 20:59:19 on dist-test-slave-1dms"
server_key: "f645b2898f8b9025644ae7aebd024f5c"
server_key_iv: "54ed70a445ad4d76473e5a4bf322a322"
server_key_version: "encryptionkey@0"
I20260929 20:59:19.813628 768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:59:19.817149 783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:19.818475 768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:59:19.818660 768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-2/wal
uuid: "43bd71fcd46545adb6281ee6b06d839e"
format_stamp: "Formatted at 2026-09-29 20:59:19 on dist-test-slave-1dms"
server_key: "f645b2898f8b9025644ae7aebd024f5c"
server_key_iv: "54ed70a445ad4d76473e5a4bf322a322"
server_key_version: "encryptionkey@0"
I20260929 20:59:19.818845 768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:19.845423 768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:19.849318 768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:19.849611 768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:19.850541 768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:59:19.851804 768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:59:19.851886 768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:19.852057 768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:59:19.852108 768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:19.886874 768 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42471
I20260929 20:59:19.886916 895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42471 every 8 connection(s)
I20260929 20:59:19.888167 768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:59:19.894822 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 768
I20260929 20:59:19.895016 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790715556572573-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:59:19.895411 388 external_mini_cluster.cc:1428] Setting key dc6f98a3a5a1ba0f4e60cd8497286576
Sep 29 20:59:19 dist-test-slave-1dms krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:59:19.909021 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:19.890705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46387 (local address 127.0.97.62:43915)
0929 20:59:19.890889 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:19.890893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:19.891945 (+ 1052us) server_negotiation.cc:407] Connection header received
0929 20:59:19.893843 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:19.893848 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:19.893938 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:19.894046 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:19.895630 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:19.896730 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:19.898422 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:19.898930 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:19.899021 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:19.902577 (+ 3556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:19.902584 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:19.902588 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:19.902625 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:19.906448 (+ 3823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:19.907136 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:19.907142 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:19.907145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:19.907273 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:19.907623 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:19.907626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:19.907628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:19.907884 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:19.907986 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:19.908559 (+ 573us) server_negotiation.cc:299] Negotiation successful
0929 20:59:19.908726 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 20:59:19.910075 898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:19.891121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43915 (local address 127.0.97.3:46387)
0929 20:59:19.891791 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:19.891828 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:19.893574 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:19.894283 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:19.894291 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:19.894789 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:19.895421 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:19.895438 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:19.896890 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:19.896897 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:19.898297 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:19.898311 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:19.899166 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:19.899176 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:19.899403 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:19.900035 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:59:19.900073 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:19.902379 (+ 2306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:19.906602 (+ 4223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:19.906608 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:19.906619 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:19.907004 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:19.907366 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:19.907369 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:19.907371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:19.907530 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:19.907984 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:19.907989 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:19.908262 (+ 273us) client_negotiation.cc:769] Sending connection context
0929 20:59:19.908464 (+ 202us) client_negotiation.cc:240] Negotiation successful
0929 20:59:19.908797 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":260,"threads_started":1}
I20260929 20:59:19.911504 896 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43915
I20260929 20:59:19.911820 896 heartbeater.cc:464] Registering TS with master...
I20260929 20:59:19.912496 896 heartbeater.cc:511] Master 127.0.97.62:43915 requested a full tablet report, sending...
I20260929 20:59:19.913858 440 ts_manager.cc:194] Registered new tserver with Master: 43bd71fcd46545adb6281ee6b06d839e (127.0.97.3:42471)
I20260929 20:59:19.914651 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46387
I20260929 20:59:19.924376 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:59:19.933290 629 heartbeater.cc:503] Master 127.0.97.62:43915 was elected leader, sending a full tablet report...
Sep 29 20:59:19 dist-test-slave-1dms krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715559, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 20:59:19 dist-test-slave-1dms krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715559, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:59:19.970062 902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:19.954033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58836 (local address 127.0.97.1:45697)
0929 20:59:19.954674 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:19.954686 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:19.955119 (+ 433us) server_negotiation.cc:407] Connection header received
0929 20:59:19.955329 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:19.955346 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:19.955575 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:19.955755 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:19.957684 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:19.958767 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:19.960054 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:19.960639 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:19.960781 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:19.964923 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:19.964935 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:19.964950 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:19.964984 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:19.967611 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:19.968208 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:19.968214 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:19.968216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:19.968326 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:19.968678 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:19.968682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:19.968685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:19.969104 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:19.969347 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:19.969585 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 20:59:19.969779 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":290,"threads_started":1}
W20260929 20:59:19.971417 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58836
Sep 29 20:59:19 dist-test-slave-1dms krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 20:59:20 dist-test-slave-1dms krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:59:20.015736 902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:20.001802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58842 (local address 127.0.97.1:45697)
0929 20:59:20.002035 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:20.002040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:20.002137 (+ 97us) server_negotiation.cc:407] Connection header received
0929 20:59:20.002296 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:20.002299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:20.002389 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:20.002478 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:20.003858 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:20.004538 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:20.005836 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:20.006254 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:20.006352 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:20.010137 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:20.010142 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:20.010145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:20.010281 (+ 136us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:20.013390 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:20.014069 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:20.014074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:20.014076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:20.014198 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:20.014544 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:20.014548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:20.014550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:20.014807 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:20.014977 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:20.015241 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 20:59:20.015393 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 20:59:20.017401 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 501
I20260929 20:59:20.030652 902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:20.018324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58852 (local address 127.0.97.1:45697)
0929 20:59:20.018500 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:20.018508 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:20.018850 (+ 342us) server_negotiation.cc:407] Connection header received
0929 20:59:20.019032 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:20.019038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:20.019113 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:20.019211 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:20.020346 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:20.021142 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:20.022668 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:20.023173 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:20.023290 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:20.025564 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:20.025569 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:20.025594 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:20.025622 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:20.028419 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:20.029054 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:20.029058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:20.029060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:20.029188 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:20.029471 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:20.029474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:20.029475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:20.029723 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:20.029902 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:20.030122 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 20:59:20.030284 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 20:59:20.031447 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:59:20.115875 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 501
I20260929 20:59:20.132145 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 635
Sep 29 20:59:20 dist-test-slave-1dms krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:59:20.147861 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:20.133062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36862 (local address 127.0.97.2:38829)
0929 20:59:20.133675 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:20.133682 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:20.133708 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:59:20.133772 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:20.133778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:20.133979 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:20.134186 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:20.135315 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:20.136332 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:20.137662 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:20.138299 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:20.138394 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:20.142230 (+ 3836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:20.142248 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:20.142262 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:20.142299 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:20.145382 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:20.146020 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:20.146026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:20.146028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:20.146137 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:20.146465 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:20.146468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:20.146470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:20.146774 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:20.146989 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:20.147279 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 20:59:20.147463 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":333,"threads_started":1}
I20260929 20:59:20.149115 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:59:20.230792 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 635
I20260929 20:59:20.253360 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 768
Sep 29 20:59:20 dist-test-slave-1dms krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:59:20.268667 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:20.254374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43814 (local address 127.0.97.3:42471)
0929 20:59:20.254962 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:20.254969 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:20.254995 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:59:20.255049 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:20.255056 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:20.255278 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:20.255400 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:20.256599 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:20.257656 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:20.258842 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:20.259461 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:20.259564 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:20.263604 (+ 4040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:20.263614 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:20.263628 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:20.263660 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:20.266453 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:20.267023 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:20.267030 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:20.267032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:20.267150 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:20.267450 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:20.267454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:20.267456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:20.267774 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:20.267997 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:20.268206 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 20:59:20.268395 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":305,"threads_started":1}
I20260929 20:59:20.270097 891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:59:20.352752 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 768
I20260929 20:59:20.369064 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 408
I20260929 20:59:20.370235 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:59:20.472903 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 408
2026-09-29T20:59:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3867 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:59:20 dist-test-slave-1dms krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:59:20 dist-test-slave-1dms krb5kdc[917](info): set up 2 sockets
Sep 29 20:59:20 dist-test-slave-1dms krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260929 20:59:21.544716 388 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.005s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 20:59:21 dist-test-slave-1dms krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:59:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:59:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:59:21.710680 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33605
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33605
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:59:22.015818 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:22.016153 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:22.016332 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:22.023504 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:59:22.023662 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:22.023748 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:22.023833 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:59:22.023916 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:59:22.034214 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:22.035579 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:22.037002 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:22.048659 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:22.048861 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:22.048775 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:22.049779 930 server_base.cc:1034] running on GCE node
I20260929 20:59:22.050552 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:22.051920 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:22.053138 930 hybrid_clock.cc:648] HybridClock initialized: now 1790715562053098 us; error 48 us; skew 500 ppm
I20260929 20:59:22.053503 930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:59:22 dist-test-slave-1dms krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:59:22.056387 930 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:59:22.057816 930 webserver.cc:460] Webserver started at http://127.0.97.62:40117/ using document root <none> and password file <none>
I20260929 20:59:22.058431 930 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:22.058524 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:22.058774 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:22.061560 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "5925643af0c54d16acd2729a8541734e"
format_stamp: "Formatted at 2026-09-29 20:59:22 on dist-test-slave-1dms"
server_key: "65aeb0f22fd3bf9ce6a436b009b029e6"
server_key_iv: "c9bbf8dee0f17a1ba52bf75ffdac0f78"
server_key_version: "encryptionkey@0"
I20260929 20:59:22.062225 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "5925643af0c54d16acd2729a8541734e"
format_stamp: "Formatted at 2026-09-29 20:59:22 on dist-test-slave-1dms"
server_key: "65aeb0f22fd3bf9ce6a436b009b029e6"
server_key_iv: "c9bbf8dee0f17a1ba52bf75ffdac0f78"
server_key_version: "encryptionkey@0"
I20260929 20:59:22.066998 930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 20:59:22.070204 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:22.071447 930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:59:22.071616 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "5925643af0c54d16acd2729a8541734e"
format_stamp: "Formatted at 2026-09-29 20:59:22 on dist-test-slave-1dms"
server_key: "65aeb0f22fd3bf9ce6a436b009b029e6"
server_key_iv: "c9bbf8dee0f17a1ba52bf75ffdac0f78"
server_key_version: "encryptionkey@0"
I20260929 20:59:22.071815 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:22.086792 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:22.090534 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:22.090822 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:22.108402 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33605
I20260929 20:59:22.108430 996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33605 every 8 connection(s)
I20260929 20:59:22.109738 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:59:22.113164 997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:22.114868 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 930
I20260929 20:59:22.115085 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 20:59:22.115708 388 external_mini_cluster.cc:1428] Setting key 4f849ad805f995b6cc8e1c9a239a03cc
I20260929 20:59:22.125602 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5925643af0c54d16acd2729a8541734e: Bootstrap starting.
I20260929 20:59:22.129084 997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5925643af0c54d16acd2729a8541734e: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:22.130023 997 log.cc:824] T 00000000000000000000000000000000 P 5925643af0c54d16acd2729a8541734e: Log is configured to *not* fsync() on all Append() calls
Sep 29 20:59:22 dist-test-slave-1dms krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:59:22.133383 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5925643af0c54d16acd2729a8541734e: No bootstrap required, opened a new log
I20260929 20:59:22.138742 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:22.117684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56554 (local address 127.0.97.62:33605)
0929 20:59:22.118482 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:22.118496 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:22.118542 (+ 46us) server_negotiation.cc:407] Connection header received
0929 20:59:22.120822 (+ 2280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:22.120879 (+ 57us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:22.121337 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:22.121778 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:22.122951 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:22.124944 (+ 1993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:22.126214 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:22.127240 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:22.127432 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:22.131538 (+ 4106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:22.131547 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:22.131566 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:22.131609 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:22.134787 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:22.135550 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:22.135555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:22.135561 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:22.135720 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:22.136091 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:22.136094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:22.136096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:22.136591 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:22.136824 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:22.137341 (+ 517us) server_negotiation.cc:299] Negotiation successful
0929 20:59:22.137701 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":305,"threads_started":1}
I20260929 20:59:22.141798 997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5925643af0c54d16acd2729a8541734e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5925643af0c54d16acd2729a8541734e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33605 } }
I20260929 20:59:22.142144 997 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5925643af0c54d16acd2729a8541734e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:22.142254 997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5925643af0c54d16acd2729a8541734e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5925643af0c54d16acd2729a8541734e, State: Initialized, Role: FOLLOWER
I20260929 20:59:22.142771 997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5925643af0c54d16acd2729a8541734e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5925643af0c54d16acd2729a8541734e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33605 } }
I20260929 20:59:22.143011 997 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5925643af0c54d16acd2729a8541734e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:59:22.143149 997 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5925643af0c54d16acd2729a8541734e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:59:22.143321 997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5925643af0c54d16acd2729a8541734e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:22.145560 997 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5925643af0c54d16acd2729a8541734e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5925643af0c54d16acd2729a8541734e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33605 } }
I20260929 20:59:22.146032 997 leader_election.cc:302] T 00000000000000000000000000000000 P 5925643af0c54d16acd2729a8541734e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5925643af0c54d16acd2729a8541734e; no voters:
I20260929 20:59:22.146564 997 leader_election.cc:288] T 00000000000000000000000000000000 P 5925643af0c54d16acd2729a8541734e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:59:22.146692 1002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5925643af0c54d16acd2729a8541734e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:59:22.147353 1002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5925643af0c54d16acd2729a8541734e [term 1 LEADER]: Becoming Leader. State: Replica: 5925643af0c54d16acd2729a8541734e, State: Running, Role: LEADER
I20260929 20:59:22.147878 1002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5925643af0c54d16acd2729a8541734e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5925643af0c54d16acd2729a8541734e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33605 } }
I20260929 20:59:22.148553 997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5925643af0c54d16acd2729a8541734e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:59:22.150612 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5925643af0c54d16acd2729a8541734e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5925643af0c54d16acd2729a8541734e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5925643af0c54d16acd2729a8541734e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33605 } } }
I20260929 20:59:22.151100 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5925643af0c54d16acd2729a8541734e [sys.catalog]: This master's current role is: LEADER
I20260929 20:59:22.150596 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5925643af0c54d16acd2729a8541734e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5925643af0c54d16acd2729a8541734e. Latest consensus state: current_term: 1 leader_uuid: "5925643af0c54d16acd2729a8541734e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5925643af0c54d16acd2729a8541734e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33605 } } }
I20260929 20:59:22.151775 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5925643af0c54d16acd2729a8541734e [sys.catalog]: This master's current role is: LEADER
I20260929 20:59:22.153669 1008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:59:22.158233 1008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:59:22.168320 1008 catalog_manager.cc:1347] Generated new cluster ID: 26bfed32ae0148a1ae9495521a879268
I20260929 20:59:22.168442 1008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:59:22.193467 1008 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:59:22.194830 1008 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:59:22.207466 1008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5925643af0c54d16acd2729a8541734e: Generated new TSK 0
I20260929 20:59:22.208300 1008 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:59:22.253304 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33605
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:59:22.612102 1023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:22.612545 1023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:22.612723 1023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:22.620898 1023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:22.621026 1023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:22.621213 1023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:59:22.633788 1023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33605
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:22.635255 1023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:22.636818 1023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:22.650126 1028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:22.650180 1031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:22.650143 1029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:22.651304 1023 server_base.cc:1034] running on GCE node
I20260929 20:59:22.651950 1023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:22.652904 1023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:22.654098 1023 hybrid_clock.cc:648] HybridClock initialized: now 1790715562654057 us; error 33 us; skew 500 ppm
I20260929 20:59:22.654407 1023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:59:22 dist-test-slave-1dms krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:59:22.657331 1023 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:59:22.658799 1023 webserver.cc:460] Webserver started at http://127.0.97.1:44861/ using document root <none> and password file <none>
I20260929 20:59:22.659523 1023 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:22.659646 1023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:22.659963 1023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:22.662797 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-0/data/instance:
uuid: "0d1b6ffb6ddf40c0ba5a36bba554517d"
format_stamp: "Formatted at 2026-09-29 20:59:22 on dist-test-slave-1dms"
server_key: "bc0e805339da1dea2a3082c7d5ef7f02"
server_key_iv: "abafaf5905d924371e84bb55ae5697d5"
server_key_version: "encryptionkey@0"
I20260929 20:59:22.663551 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0d1b6ffb6ddf40c0ba5a36bba554517d"
format_stamp: "Formatted at 2026-09-29 20:59:22 on dist-test-slave-1dms"
server_key: "bc0e805339da1dea2a3082c7d5ef7f02"
server_key_iv: "abafaf5905d924371e84bb55ae5697d5"
server_key_version: "encryptionkey@0"
I20260929 20:59:22.668473 1023 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 20:59:22.671765 1038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:22.673072 1023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:59:22.673228 1023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-0/wal
uuid: "0d1b6ffb6ddf40c0ba5a36bba554517d"
format_stamp: "Formatted at 2026-09-29 20:59:22 on dist-test-slave-1dms"
server_key: "bc0e805339da1dea2a3082c7d5ef7f02"
server_key_iv: "abafaf5905d924371e84bb55ae5697d5"
server_key_version: "encryptionkey@0"
I20260929 20:59:22.673429 1023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:22.702266 1023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:22.706278 1023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:22.706766 1023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:22.707811 1023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:59:22.709125 1023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:59:22.709193 1023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:22.709358 1023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:59:22.709434 1023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:22.741637 1023 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40953
I20260929 20:59:22.741683 1150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40953 every 8 connection(s)
I20260929 20:59:22.742909 1023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:59:22.752599 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 1023
I20260929 20:59:22.752818 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:59:22.753572 388 external_mini_cluster.cc:1428] Setting key 9624aa7913f037c0001aa8edffc55528
Sep 29 20:59:22 dist-test-slave-1dms krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:59:22.762744 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:22.745375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35303 (local address 127.0.97.62:33605)
0929 20:59:22.745826 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:22.745830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:22.746629 (+ 799us) server_negotiation.cc:407] Connection header received
0929 20:59:22.748446 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:22.748451 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:22.748536 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:22.748654 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:22.750178 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:22.750897 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:22.752501 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:22.753146 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:22.753257 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:22.757019 (+ 3762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:22.757024 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:22.757027 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:22.757056 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:22.759598 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:22.760512 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:22.760516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:22.760518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:22.760625 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:22.761115 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:22.761118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:22.761120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:22.761359 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:22.761465 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:22.762327 (+ 862us) server_negotiation.cc:299] Negotiation successful
0929 20:59:22.762481 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":255,"threads_started":1}
I20260929 20:59:22.764217 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:22.745856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33605 (local address 127.0.97.1:35303)
0929 20:59:22.746482 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:22.746517 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:22.748119 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:22.748907 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:22.748916 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:22.749483 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:22.750016 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:22.750032 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:22.751045 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:22.751052 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:22.752362 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:22.752377 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:22.753382 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:22.753396 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:22.753707 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:22.754473 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:59:22.754503 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:22.756748 (+ 2245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:22.759759 (+ 3011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:22.759768 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:22.759788 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:22.760397 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:22.760729 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:22.760737 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:22.760743 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:22.761010 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:22.761479 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:22.761490 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:22.761930 (+ 440us) client_negotiation.cc:769] Sending connection context
0929 20:59:22.762225 (+ 295us) client_negotiation.cc:240] Negotiation successful
0929 20:59:22.762801 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":254,"threads_started":1}
I20260929 20:59:22.765797 1151 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33605
I20260929 20:59:22.766127 1151 heartbeater.cc:464] Registering TS with master...
I20260929 20:59:22.767011 1151 heartbeater.cc:511] Master 127.0.97.62:33605 requested a full tablet report, sending...
I20260929 20:59:22.769068 962 ts_manager.cc:194] Registered new tserver with Master: 0d1b6ffb6ddf40c0ba5a36bba554517d (127.0.97.1:40953)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 20:59:22.770515 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35303
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:59:22.793573 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33605
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:59:23.102280 1157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:23.102581 1157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:23.102739 1157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:23.110050 1157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:23.110183 1157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:23.110368 1157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:59:23.119771 1157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33605
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:23.121073 1157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:23.122339 1157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:23.135377 1163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:23.135377 1165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:23.135411 1162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:23.136042 1157 server_base.cc:1034] running on GCE node
I20260929 20:59:23.136732 1157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:23.137645 1157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:23.138875 1157 hybrid_clock.cc:648] HybridClock initialized: now 1790715563138837 us; error 49 us; skew 500 ppm
I20260929 20:59:23.139185 1157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:59:23 dist-test-slave-1dms krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:59:23.142292 1157 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:59:23.143666 1157 webserver.cc:460] Webserver started at http://127.0.97.2:36771/ using document root <none> and password file <none>
I20260929 20:59:23.144275 1157 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:23.144368 1157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:23.144621 1157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:23.147477 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-1/data/instance:
uuid: "2bf99c0628e545b98d875c4fed00fe87"
format_stamp: "Formatted at 2026-09-29 20:59:23 on dist-test-slave-1dms"
server_key: "62a156f42069e78f7c34a3c3ee7ec10a"
server_key_iv: "9cdb6c6948dee40b08274982882f4cee"
server_key_version: "encryptionkey@0"
I20260929 20:59:23.148151 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2bf99c0628e545b98d875c4fed00fe87"
format_stamp: "Formatted at 2026-09-29 20:59:23 on dist-test-slave-1dms"
server_key: "62a156f42069e78f7c34a3c3ee7ec10a"
server_key_iv: "9cdb6c6948dee40b08274982882f4cee"
server_key_version: "encryptionkey@0"
I20260929 20:59:23.153115 1157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260929 20:59:23.156219 1172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:23.157536 1157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:59:23.157734 1157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-1/wal
uuid: "2bf99c0628e545b98d875c4fed00fe87"
format_stamp: "Formatted at 2026-09-29 20:59:23 on dist-test-slave-1dms"
server_key: "62a156f42069e78f7c34a3c3ee7ec10a"
server_key_iv: "9cdb6c6948dee40b08274982882f4cee"
server_key_version: "encryptionkey@0"
I20260929 20:59:23.157917 1157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:23.181771 1157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:23.185447 1157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:23.185731 1157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:23.186626 1157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:59:23.187886 1157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:59:23.187952 1157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:23.188104 1157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:59:23.188162 1157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:23.219521 1157 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37511
I20260929 20:59:23.219575 1284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37511 every 8 connection(s)
I20260929 20:59:23.220728 1157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:59:23.228134 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 1157
I20260929 20:59:23.228327 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:59:23.228724 388 external_mini_cluster.cc:1428] Setting key 488b7cde0a43cda5561e89e9c454eb20
Sep 29 20:59:23 dist-test-slave-1dms krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 20:59:23.246850 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:23.223145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50821 (local address 127.0.97.62:33605)
0929 20:59:23.223315 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:23.223319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:23.224406 (+ 1087us) server_negotiation.cc:407] Connection header received
0929 20:59:23.226215 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:23.226220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:23.226312 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:23.226421 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:23.227756 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:23.228549 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:23.230502 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:23.231097 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:23.231245 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:23.240158 (+ 8913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:23.240163 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:23.240166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:23.240197 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:23.242822 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:23.243895 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:23.243903 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:23.243908 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:23.244096 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:23.244655 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:23.244661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:23.244664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:23.245072 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:23.245571 (+ 499us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:23.246405 (+ 834us) server_negotiation.cc:299] Negotiation successful
0929 20:59:23.246618 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 20:59:23.247946 1287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:23.223604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33605 (local address 127.0.97.2:50821)
0929 20:59:23.224256 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:23.224294 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:23.225969 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:23.226628 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:23.226636 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:23.227085 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:23.227595 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:23.227610 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:23.228681 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:23.228691 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:23.230376 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:23.230390 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:23.231293 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:23.231299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:23.231478 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:23.237242 (+ 5764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:59:23.237284 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:23.239944 (+ 2660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:23.242977 (+ 3033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:23.243019 (+ 42us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:23.243039 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:23.243633 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:23.244238 (+ 605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:23.244242 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:23.244246 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:23.244490 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:23.245259 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:23.245267 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:23.245703 (+ 436us) client_negotiation.cc:769] Sending connection context
0929 20:59:23.246008 (+ 305us) client_negotiation.cc:240] Negotiation successful
0929 20:59:23.246596 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":261,"threads_started":1}
I20260929 20:59:23.249379 1285 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33605
I20260929 20:59:23.249667 1285 heartbeater.cc:464] Registering TS with master...
I20260929 20:59:23.250349 1285 heartbeater.cc:511] Master 127.0.97.62:33605 requested a full tablet report, sending...
I20260929 20:59:23.251698 962 ts_manager.cc:194] Registered new tserver with Master: 2bf99c0628e545b98d875c4fed00fe87 (127.0.97.2:37511)
I20260929 20:59:23.252753 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50821
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:59:23.268642 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33605
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:59:23.581926 1290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:23.582199 1290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:23.582301 1290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:23.589592 1290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:23.589695 1290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:23.589807 1290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:59:23.599588 1290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33605
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:23.600823 1290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:23.602172 1290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:23.615247 1295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:23.615270 1296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:23.615510 1298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:23.616014 1290 server_base.cc:1034] running on GCE node
I20260929 20:59:23.616568 1290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:23.617506 1290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:23.618690 1290 hybrid_clock.cc:648] HybridClock initialized: now 1790715563618646 us; error 43 us; skew 500 ppm
I20260929 20:59:23.619010 1290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:59:23 dist-test-slave-1dms krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:59:23.622074 1290 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:59:23.623489 1290 webserver.cc:460] Webserver started at http://127.0.97.3:35443/ using document root <none> and password file <none>
I20260929 20:59:23.624161 1290 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:23.624307 1290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:23.624610 1290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:23.627624 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-2/data/instance:
uuid: "7a697ee8d2ab4b85aa481c33d6698b10"
format_stamp: "Formatted at 2026-09-29 20:59:23 on dist-test-slave-1dms"
server_key: "5e6b177906fdcf2ec318f3c47f4996ba"
server_key_iv: "14314279b5d0d68a45e1eaac06a5c93d"
server_key_version: "encryptionkey@0"
I20260929 20:59:23.628388 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7a697ee8d2ab4b85aa481c33d6698b10"
format_stamp: "Formatted at 2026-09-29 20:59:23 on dist-test-slave-1dms"
server_key: "5e6b177906fdcf2ec318f3c47f4996ba"
server_key_iv: "14314279b5d0d68a45e1eaac06a5c93d"
server_key_version: "encryptionkey@0"
I20260929 20:59:23.633332 1290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260929 20:59:23.636516 1305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:23.637784 1290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:59:23.637939 1290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-2/wal
uuid: "7a697ee8d2ab4b85aa481c33d6698b10"
format_stamp: "Formatted at 2026-09-29 20:59:23 on dist-test-slave-1dms"
server_key: "5e6b177906fdcf2ec318f3c47f4996ba"
server_key_iv: "14314279b5d0d68a45e1eaac06a5c93d"
server_key_version: "encryptionkey@0"
I20260929 20:59:23.638116 1290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:23.661592 1290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:23.665419 1290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:23.665710 1290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:23.666656 1290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:59:23.668025 1290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:59:23.668110 1290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:23.668293 1290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:59:23.668342 1290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:23.700604 1290 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34823
I20260929 20:59:23.700645 1417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34823 every 8 connection(s)
I20260929 20:59:23.701859 1290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:59:23.703392 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 1290
I20260929 20:59:23.703574 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790715556572573-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:59:23.704169 388 external_mini_cluster.cc:1428] Setting key 74413d532cd7e504e932d9ee5563bc90
Sep 29 20:59:23 dist-test-slave-1dms krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:59:23.721201 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:23.704664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43117 (local address 127.0.97.62:33605)
0929 20:59:23.704895 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:23.704900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:23.706674 (+ 1774us) server_negotiation.cc:407] Connection header received
0929 20:59:23.708454 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:23.708458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:23.708521 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:23.708601 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:23.710016 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:23.710734 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:23.712154 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:23.712523 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:23.712607 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:23.715675 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:23.715680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:23.715683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:23.715712 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:23.718419 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:23.719087 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:23.719096 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:23.719098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:23.719208 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:23.719604 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:23.719608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:23.719612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:23.719853 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:23.719992 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:23.720643 (+ 651us) server_negotiation.cc:299] Negotiation successful
0929 20:59:23.720873 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:59:23.722035 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:23.705242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33605 (local address 127.0.97.3:43117)
0929 20:59:23.706502 (+ 1260us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:23.706558 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:23.708210 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:23.708803 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:23.708810 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:23.709339 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:23.709879 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:23.709896 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:23.710865 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:23.710868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:23.712044 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:23.712054 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:23.712686 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:23.712690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:23.712939 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:23.713501 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:59:23.713521 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:23.715500 (+ 1979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:23.718547 (+ 3047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:23.718555 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:23.718569 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:23.718961 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:23.719316 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:23.719320 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:23.719324 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:23.719492 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:23.719988 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:23.719994 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:23.720293 (+ 299us) client_negotiation.cc:769] Sending connection context
0929 20:59:23.720533 (+ 240us) client_negotiation.cc:240] Negotiation successful
0929 20:59:23.721031 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1006,"thread_start_us":370,"threads_started":1}
I20260929 20:59:23.723330 1418 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33605
I20260929 20:59:23.723562 1418 heartbeater.cc:464] Registering TS with master...
I20260929 20:59:23.724151 1418 heartbeater.cc:511] Master 127.0.97.62:33605 requested a full tablet report, sending...
I20260929 20:59:23.725448 962 ts_manager.cc:194] Registered new tserver with Master: 7a697ee8d2ab4b85aa481c33d6698b10 (127.0.97.3:34823)
I20260929 20:59:23.726167 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:43117
I20260929 20:59:23.731062 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:59:23.749341 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:23.737735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56560 (local address 127.0.97.62:33605)
0929 20:59:23.737877 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:23.737881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:23.737896 (+ 15us) server_negotiation.cc:407] Connection header received
0929 20:59:23.738017 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:23.738020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:23.738072 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:23.738143 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:23.739397 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:23.740052 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:23.741431 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:23.741796 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:23.741882 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:23.744599 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:23.744604 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:23.744607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:23.744634 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:23.747158 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:23.747677 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:23.747681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:23.747683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:23.747794 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:23.748077 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:23.748080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:23.748082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:23.748325 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:23.748585 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:23.748825 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 20:59:23.749016 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 20:59:23.754544 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:59:23.761634 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:59:23.773852 1151 heartbeater.cc:503] Master 127.0.97.62:33605 was elected leader, sending a full tablet report...
I20260929 20:59:23.785907 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:23.773895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37511 (local address 127.0.0.1:39674)
0929 20:59:23.776782 (+ 2887us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:23.776801 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:23.777003 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:23.777908 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:23.777918 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:23.777952 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:59:23.778513 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:23.778526 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:23.780422 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:23.780427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:23.782814 (+ 2387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:23.782828 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:23.784907 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:23.784913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:23.785119 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:23.785168 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 20:59:23.785291 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 20:59:23.785459 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2760,"mutex_wait_us":54,"thread_start_us":2631,"threads_started":1}
I20260929 20:59:23.786805 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:23.774152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39674 (local address 127.0.97.2:37511)
0929 20:59:23.776411 (+ 2259us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:23.776419 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:23.777014 (+ 595us) server_negotiation.cc:407] Connection header received
0929 20:59:23.777208 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:23.777216 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:23.777464 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:23.777640 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:59:23.778689 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:23.780264 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:23.782993 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:23.784425 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:23.784544 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:23.784762 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:23.786322 (+ 1560us) server_negotiation.cc:299] Negotiation successful
0929 20:59:23.786490 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2116,"thread_start_us":2611,"threads_started":1}
I20260929 20:59:23.787122 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:23.772485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40953 (local address 127.0.0.1:48650)
0929 20:59:23.773261 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:23.773306 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:23.773461 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:23.777989 (+ 4528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:23.777995 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:23.778022 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:59:23.778440 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:23.778458 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:23.781491 (+ 3033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:23.781501 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:23.783586 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:23.783599 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:23.786329 (+ 2730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:23.786336 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:23.786503 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:23.786532 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 20:59:23.786598 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 20:59:23.786713 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":1,"thread_start_us":515,"threads_started":1}
I20260929 20:59:23.788544 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:23.772876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48650 (local address 127.0.97.1:40953)
0929 20:59:23.777456 (+ 4580us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:23.777464 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:23.777490 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:59:23.777572 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:23.777581 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:23.777804 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:23.778645 (+ 841us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:59:23.778854 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:23.781346 (+ 2492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:23.784671 (+ 3325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:23.786109 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:23.787318 (+ 1209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:23.787561 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:23.787636 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 20:59:23.788204 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4436,"thread_start_us":4292,"threads_started":1}
I20260929 20:59:23.792230 1086 tablet_service.cc:1467] Processing CreateTablet for tablet d4710222a5a345f7990f1682699974f9 (DEFAULT_TABLE table=test-table [id=5904e1542690475b98dbf0eaf4709ec1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:59:23.792277 1220 tablet_service.cc:1467] Processing CreateTablet for tablet d4710222a5a345f7990f1682699974f9 (DEFAULT_TABLE table=test-table [id=5904e1542690475b98dbf0eaf4709ec1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:59:23.793501 1220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4710222a5a345f7990f1682699974f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:23.793521 1086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4710222a5a345f7990f1682699974f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:23.795452 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:23.777495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34823 (local address 127.0.0.1:35820)
0929 20:59:23.787100 (+ 9605us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:23.787122 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:23.787212 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:23.789088 (+ 1876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:23.789092 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:23.789121 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:59:23.789286 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:23.789296 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:23.791358 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:23.791364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:23.793084 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:23.793107 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:23.794744 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:23.794750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:23.794917 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:23.794972 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 20:59:23.795070 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 20:59:23.795202 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9489,"thread_start_us":11048,"threads_started":1}
I20260929 20:59:23.795580 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:23.777283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35820 (local address 127.0.97.3:34823)
0929 20:59:23.788454 (+ 11171us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:23.788463 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:23.788495 (+ 32us) server_negotiation.cc:407] Connection header received
0929 20:59:23.788593 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:23.788601 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:23.788881 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:23.789447 (+ 566us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:59:23.789705 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:23.791206 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:23.793256 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:23.794577 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:23.794722 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:23.794928 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:23.795123 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 20:59:23.795277 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11003,"thread_start_us":10847,"threads_started":1}
I20260929 20:59:23.799304 1353 tablet_service.cc:1467] Processing CreateTablet for tablet d4710222a5a345f7990f1682699974f9 (DEFAULT_TABLE table=test-table [id=5904e1542690475b98dbf0eaf4709ec1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:59:23.800644 1353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4710222a5a345f7990f1682699974f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:23.804885 1436 tablet_bootstrap.cc:492] T d4710222a5a345f7990f1682699974f9 P 2bf99c0628e545b98d875c4fed00fe87: Bootstrap starting.
I20260929 20:59:23.807670 1437 tablet_bootstrap.cc:492] T d4710222a5a345f7990f1682699974f9 P 0d1b6ffb6ddf40c0ba5a36bba554517d: Bootstrap starting.
I20260929 20:59:23.808320 1436 tablet_bootstrap.cc:654] T d4710222a5a345f7990f1682699974f9 P 2bf99c0628e545b98d875c4fed00fe87: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:23.809578 1436 log.cc:824] T d4710222a5a345f7990f1682699974f9 P 2bf99c0628e545b98d875c4fed00fe87: Log is configured to *not* fsync() on all Append() calls
I20260929 20:59:23.810964 1437 tablet_bootstrap.cc:654] T d4710222a5a345f7990f1682699974f9 P 0d1b6ffb6ddf40c0ba5a36bba554517d: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:23.812368 1437 log.cc:824] T d4710222a5a345f7990f1682699974f9 P 0d1b6ffb6ddf40c0ba5a36bba554517d: Log is configured to *not* fsync() on all Append() calls
I20260929 20:59:23.812675 1436 tablet_bootstrap.cc:492] T d4710222a5a345f7990f1682699974f9 P 2bf99c0628e545b98d875c4fed00fe87: No bootstrap required, opened a new log
I20260929 20:59:23.813011 1436 ts_tablet_manager.cc:1397] T d4710222a5a345f7990f1682699974f9 P 2bf99c0628e545b98d875c4fed00fe87: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260929 20:59:23.814857 1440 tablet_bootstrap.cc:492] T d4710222a5a345f7990f1682699974f9 P 7a697ee8d2ab4b85aa481c33d6698b10: Bootstrap starting.
I20260929 20:59:23.817879 1440 tablet_bootstrap.cc:654] T d4710222a5a345f7990f1682699974f9 P 7a697ee8d2ab4b85aa481c33d6698b10: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:23.818769 1440 log.cc:824] T d4710222a5a345f7990f1682699974f9 P 7a697ee8d2ab4b85aa481c33d6698b10: Log is configured to *not* fsync() on all Append() calls
I20260929 20:59:23.819332 1437 tablet_bootstrap.cc:492] T d4710222a5a345f7990f1682699974f9 P 0d1b6ffb6ddf40c0ba5a36bba554517d: No bootstrap required, opened a new log
I20260929 20:59:23.819597 1437 ts_tablet_manager.cc:1397] T d4710222a5a345f7990f1682699974f9 P 0d1b6ffb6ddf40c0ba5a36bba554517d: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260929 20:59:23.821135 1440 tablet_bootstrap.cc:492] T d4710222a5a345f7990f1682699974f9 P 7a697ee8d2ab4b85aa481c33d6698b10: No bootstrap required, opened a new log
I20260929 20:59:23.821372 1440 ts_tablet_manager.cc:1397] T d4710222a5a345f7990f1682699974f9 P 7a697ee8d2ab4b85aa481c33d6698b10: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 20:59:23.824376 1436 raft_consensus.cc:348] T d4710222a5a345f7990f1682699974f9 P 2bf99c0628e545b98d875c4fed00fe87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d1b6ffb6ddf40c0ba5a36bba554517d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40953 } } peers { permanent_uuid: "2bf99c0628e545b98d875c4fed00fe87" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37511 } } peers { permanent_uuid: "7a697ee8d2ab4b85aa481c33d6698b10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34823 } }
I20260929 20:59:23.824782 1436 raft_consensus.cc:374] T d4710222a5a345f7990f1682699974f9 P 2bf99c0628e545b98d875c4fed00fe87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:23.824910 1436 raft_consensus.cc:729] T d4710222a5a345f7990f1682699974f9 P 2bf99c0628e545b98d875c4fed00fe87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bf99c0628e545b98d875c4fed00fe87, State: Initialized, Role: FOLLOWER
I20260929 20:59:23.825491 1436 consensus_queue.cc:260] T d4710222a5a345f7990f1682699974f9 P 2bf99c0628e545b98d875c4fed00fe87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d1b6ffb6ddf40c0ba5a36bba554517d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40953 } } peers { permanent_uuid: "2bf99c0628e545b98d875c4fed00fe87" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37511 } } peers { permanent_uuid: "7a697ee8d2ab4b85aa481c33d6698b10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34823 } }
I20260929 20:59:23.826551 1285 heartbeater.cc:503] Master 127.0.97.62:33605 was elected leader, sending a full tablet report...
I20260929 20:59:23.827572 1436 ts_tablet_manager.cc:1428] T d4710222a5a345f7990f1682699974f9 P 2bf99c0628e545b98d875c4fed00fe87: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260929 20:59:23.829725 1437 raft_consensus.cc:348] T d4710222a5a345f7990f1682699974f9 P 0d1b6ffb6ddf40c0ba5a36bba554517d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d1b6ffb6ddf40c0ba5a36bba554517d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40953 } } peers { permanent_uuid: "2bf99c0628e545b98d875c4fed00fe87" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37511 } } peers { permanent_uuid: "7a697ee8d2ab4b85aa481c33d6698b10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34823 } }
I20260929 20:59:23.830154 1437 raft_consensus.cc:374] T d4710222a5a345f7990f1682699974f9 P 0d1b6ffb6ddf40c0ba5a36bba554517d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:23.830263 1437 raft_consensus.cc:729] T d4710222a5a345f7990f1682699974f9 P 0d1b6ffb6ddf40c0ba5a36bba554517d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d1b6ffb6ddf40c0ba5a36bba554517d, State: Initialized, Role: FOLLOWER
I20260929 20:59:23.830866 1437 consensus_queue.cc:260] T d4710222a5a345f7990f1682699974f9 P 0d1b6ffb6ddf40c0ba5a36bba554517d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d1b6ffb6ddf40c0ba5a36bba554517d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40953 } } peers { permanent_uuid: "2bf99c0628e545b98d875c4fed00fe87" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37511 } } peers { permanent_uuid: "7a697ee8d2ab4b85aa481c33d6698b10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34823 } }
I20260929 20:59:23.832548 1437 ts_tablet_manager.cc:1428] T d4710222a5a345f7990f1682699974f9 P 0d1b6ffb6ddf40c0ba5a36bba554517d: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260929 20:59:23.833514 1440 raft_consensus.cc:348] T d4710222a5a345f7990f1682699974f9 P 7a697ee8d2ab4b85aa481c33d6698b10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d1b6ffb6ddf40c0ba5a36bba554517d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40953 } } peers { permanent_uuid: "2bf99c0628e545b98d875c4fed00fe87" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37511 } } peers { permanent_uuid: "7a697ee8d2ab4b85aa481c33d6698b10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34823 } }
I20260929 20:59:23.833920 1440 raft_consensus.cc:374] T d4710222a5a345f7990f1682699974f9 P 7a697ee8d2ab4b85aa481c33d6698b10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:23.834015 1440 raft_consensus.cc:729] T d4710222a5a345f7990f1682699974f9 P 7a697ee8d2ab4b85aa481c33d6698b10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a697ee8d2ab4b85aa481c33d6698b10, State: Initialized, Role: FOLLOWER
I20260929 20:59:23.834594 1440 consensus_queue.cc:260] T d4710222a5a345f7990f1682699974f9 P 7a697ee8d2ab4b85aa481c33d6698b10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d1b6ffb6ddf40c0ba5a36bba554517d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40953 } } peers { permanent_uuid: "2bf99c0628e545b98d875c4fed00fe87" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37511 } } peers { permanent_uuid: "7a697ee8d2ab4b85aa481c33d6698b10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34823 } }
I20260929 20:59:23.835695 1418 heartbeater.cc:503] Master 127.0.97.62:33605 was elected leader, sending a full tablet report...
I20260929 20:59:23.836232 1440 ts_tablet_manager.cc:1428] T d4710222a5a345f7990f1682699974f9 P 7a697ee8d2ab4b85aa481c33d6698b10: Time spent starting tablet: real 0.015s user 0.011s sys 0.000s
W20260929 20:59:23.953444 1419 tablet.cc:2369] T d4710222a5a345f7990f1682699974f9 P 7a697ee8d2ab4b85aa481c33d6698b10: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:59:23.972609 1286 tablet.cc:2369] T d4710222a5a345f7990f1682699974f9 P 2bf99c0628e545b98d875c4fed00fe87: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:59:23.995296 1152 tablet.cc:2369] T d4710222a5a345f7990f1682699974f9 P 0d1b6ffb6ddf40c0ba5a36bba554517d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:59:24.009136 1443 raft_consensus.cc:482] T d4710222a5a345f7990f1682699974f9 P 0d1b6ffb6ddf40c0ba5a36bba554517d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:59:24.009482 1443 raft_consensus.cc:504] T d4710222a5a345f7990f1682699974f9 P 0d1b6ffb6ddf40c0ba5a36bba554517d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d1b6ffb6ddf40c0ba5a36bba554517d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40953 } } peers { permanent_uuid: "2bf99c0628e545b98d875c4fed00fe87" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37511 } } peers { permanent_uuid: "7a697ee8d2ab4b85aa481c33d6698b10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34823 } }
I20260929 20:59:24.011227 1443 leader_election.cc:288] T d4710222a5a345f7990f1682699974f9 P 0d1b6ffb6ddf40c0ba5a36bba554517d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bf99c0628e545b98d875c4fed00fe87 (127.0.97.2:37511), 7a697ee8d2ab4b85aa481c33d6698b10 (127.0.97.3:34823)
I20260929 20:59:24.017335 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:24.011667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50903 (local address 127.0.97.2:37511)
0929 20:59:24.011849 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:24.011853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:24.012332 (+ 479us) server_negotiation.cc:407] Connection header received
0929 20:59:24.012531 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:24.012537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:24.012617 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:24.012732 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:59:24.013287 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:24.013972 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:24.015640 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:24.016399 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:24.016553 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:24.016648 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:24.016969 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 20:59:24.017087 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:59:24.017452 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:24.011545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37511 (local address 127.0.97.1:50903)
0929 20:59:24.012220 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:24.012242 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:24.012381 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:24.012754 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:24.012758 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:24.012788 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:59:24.013143 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:24.013152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:24.014096 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:24.014103 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:24.015480 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:24.015491 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:24.016600 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:24.016606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:24.016780 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:24.016819 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 20:59:24.016946 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 20:59:24.017065 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":371,"threads_started":1}
I20260929 20:59:24.017931 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:24.012490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53555 (local address 127.0.97.3:34823)
0929 20:59:24.012662 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:24.012670 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:24.013064 (+ 394us) server_negotiation.cc:407] Connection header received
0929 20:59:24.013256 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:24.013260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:24.013314 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:24.013405 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:59:24.013875 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:24.014559 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:24.016198 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:24.017005 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:24.017134 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:24.017237 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:24.017561 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 20:59:24.017670 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 20:59:24.018563 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:24.012362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34823 (local address 127.0.97.1:53555)
0929 20:59:24.012966 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:24.012979 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:24.013112 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:24.013416 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:24.013420 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:24.013441 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:59:24.013727 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:24.013734 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:24.014684 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:24.014690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:24.016059 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:24.016069 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:24.017217 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:24.017225 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:24.017361 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:24.017384 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 20:59:24.018103 (+ 719us) client_negotiation.cc:240] Negotiation successful
0929 20:59:24.018242 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":279,"threads_started":1}
I20260929 20:59:24.018656 1240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4710222a5a345f7990f1682699974f9" candidate_uuid: "0d1b6ffb6ddf40c0ba5a36bba554517d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bf99c0628e545b98d875c4fed00fe87" is_pre_election: true
I20260929 20:59:24.019388 1373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4710222a5a345f7990f1682699974f9" candidate_uuid: "0d1b6ffb6ddf40c0ba5a36bba554517d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a697ee8d2ab4b85aa481c33d6698b10" is_pre_election: true
I20260929 20:59:24.019162 1240 raft_consensus.cc:2454] T d4710222a5a345f7990f1682699974f9 P 2bf99c0628e545b98d875c4fed00fe87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d1b6ffb6ddf40c0ba5a36bba554517d in term 0.
I20260929 20:59:24.019825 1373 raft_consensus.cc:2454] T d4710222a5a345f7990f1682699974f9 P 7a697ee8d2ab4b85aa481c33d6698b10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d1b6ffb6ddf40c0ba5a36bba554517d in term 0.
I20260929 20:59:24.020220 1041 leader_election.cc:302] T d4710222a5a345f7990f1682699974f9 P 0d1b6ffb6ddf40c0ba5a36bba554517d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d1b6ffb6ddf40c0ba5a36bba554517d, 2bf99c0628e545b98d875c4fed00fe87; no voters:
I20260929 20:59:24.020576 1443 raft_consensus.cc:2781] T d4710222a5a345f7990f1682699974f9 P 0d1b6ffb6ddf40c0ba5a36bba554517d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:59:24.020710 1443 raft_consensus.cc:482] T d4710222a5a345f7990f1682699974f9 P 0d1b6ffb6ddf40c0ba5a36bba554517d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:59:24.020812 1443 raft_consensus.cc:3037] T d4710222a5a345f7990f1682699974f9 P 0d1b6ffb6ddf40c0ba5a36bba554517d [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:24.022899 1443 raft_consensus.cc:504] T d4710222a5a345f7990f1682699974f9 P 0d1b6ffb6ddf40c0ba5a36bba554517d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d1b6ffb6ddf40c0ba5a36bba554517d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40953 } } peers { permanent_uuid: "2bf99c0628e545b98d875c4fed00fe87" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37511 } } peers { permanent_uuid: "7a697ee8d2ab4b85aa481c33d6698b10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34823 } }
I20260929 20:59:24.023448 1443 leader_election.cc:288] T d4710222a5a345f7990f1682699974f9 P 0d1b6ffb6ddf40c0ba5a36bba554517d [CANDIDATE]: Term 1 election: Requested vote from peers 2bf99c0628e545b98d875c4fed00fe87 (127.0.97.2:37511), 7a697ee8d2ab4b85aa481c33d6698b10 (127.0.97.3:34823)
I20260929 20:59:24.023857 1240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4710222a5a345f7990f1682699974f9" candidate_uuid: "0d1b6ffb6ddf40c0ba5a36bba554517d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bf99c0628e545b98d875c4fed00fe87"
I20260929 20:59:24.023964 1373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4710222a5a345f7990f1682699974f9" candidate_uuid: "0d1b6ffb6ddf40c0ba5a36bba554517d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a697ee8d2ab4b85aa481c33d6698b10"
I20260929 20:59:24.024063 1240 raft_consensus.cc:3037] T d4710222a5a345f7990f1682699974f9 P 2bf99c0628e545b98d875c4fed00fe87 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:24.024183 1373 raft_consensus.cc:3037] T d4710222a5a345f7990f1682699974f9 P 7a697ee8d2ab4b85aa481c33d6698b10 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:24.026232 1240 raft_consensus.cc:2454] T d4710222a5a345f7990f1682699974f9 P 2bf99c0628e545b98d875c4fed00fe87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d1b6ffb6ddf40c0ba5a36bba554517d in term 1.
I20260929 20:59:24.026345 1373 raft_consensus.cc:2454] T d4710222a5a345f7990f1682699974f9 P 7a697ee8d2ab4b85aa481c33d6698b10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d1b6ffb6ddf40c0ba5a36bba554517d in term 1.
I20260929 20:59:24.026618 1041 leader_election.cc:302] T d4710222a5a345f7990f1682699974f9 P 0d1b6ffb6ddf40c0ba5a36bba554517d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d1b6ffb6ddf40c0ba5a36bba554517d, 2bf99c0628e545b98d875c4fed00fe87; no voters:
I20260929 20:59:24.026887 1443 raft_consensus.cc:2781] T d4710222a5a345f7990f1682699974f9 P 0d1b6ffb6ddf40c0ba5a36bba554517d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:59:24.027302 1443 raft_consensus.cc:686] T d4710222a5a345f7990f1682699974f9 P 0d1b6ffb6ddf40c0ba5a36bba554517d [term 1 LEADER]: Becoming Leader. State: Replica: 0d1b6ffb6ddf40c0ba5a36bba554517d, State: Running, Role: LEADER
I20260929 20:59:24.027733 1443 consensus_queue.cc:237] T d4710222a5a345f7990f1682699974f9 P 0d1b6ffb6ddf40c0ba5a36bba554517d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d1b6ffb6ddf40c0ba5a36bba554517d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40953 } } peers { permanent_uuid: "2bf99c0628e545b98d875c4fed00fe87" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37511 } } peers { permanent_uuid: "7a697ee8d2ab4b85aa481c33d6698b10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34823 } }
I20260929 20:59:24.032207 962 catalog_manager.cc:5495] T d4710222a5a345f7990f1682699974f9 P 0d1b6ffb6ddf40c0ba5a36bba554517d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d1b6ffb6ddf40c0ba5a36bba554517d (127.0.97.1). New cstate: current_term: 1 leader_uuid: "0d1b6ffb6ddf40c0ba5a36bba554517d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d1b6ffb6ddf40c0ba5a36bba554517d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2bf99c0628e545b98d875c4fed00fe87" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a697ee8d2ab4b85aa481c33d6698b10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34823 } health_report { overall_health: UNKNOWN } } }
Sep 29 20:59:24 dist-test-slave-1dms krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:59:24.082506 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:24.069240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48658 (local address 127.0.97.1:40953)
0929 20:59:24.069499 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:24.069508 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:24.069910 (+ 402us) server_negotiation.cc:407] Connection header received
0929 20:59:24.070040 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:24.070044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:24.070105 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:24.070198 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:24.071555 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:24.072238 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:24.073415 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:24.073803 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:24.073893 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:24.077730 (+ 3837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:24.077743 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:24.077762 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:24.077808 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:24.080393 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:24.080963 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:24.080970 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:24.080972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:24.081097 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:24.081390 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:24.081392 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:24.081394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:24.081688 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:24.081902 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:24.082087 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 20:59:24.082269 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 20:59:24 dist-test-slave-1dms krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715564, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 20:59:24 dist-test-slave-1dms krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715564, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:59:24.128925 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:24.113490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48670 (local address 127.0.97.1:40953)
0929 20:59:24.113680 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:24.113688 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:24.114207 (+ 519us) server_negotiation.cc:407] Connection header received
0929 20:59:24.114364 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:24.114367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:24.114433 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:24.114532 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:24.116124 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:24.116913 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:24.118792 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:24.119184 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:24.119297 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:24.123661 (+ 4364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:24.123666 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:24.123669 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:24.123702 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:24.126665 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:24.127200 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:24.127203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:24.127205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:24.127327 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:24.127653 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:24.127655 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:24.127657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:24.127885 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:24.128083 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:24.128262 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 20:59:24.128500 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 20:59:24 dist-test-slave-1dms krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 20:59:24 dist-test-slave-1dms krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:59:24.172094 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:24.159101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48682 (local address 127.0.97.1:40953)
0929 20:59:24.159339 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:24.159346 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:24.159564 (+ 218us) server_negotiation.cc:407] Connection header received
0929 20:59:24.159735 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:24.159740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:24.159837 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:24.159922 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:24.161275 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:24.162097 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:24.163273 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:24.163675 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:24.163769 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:24.167712 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:24.167717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:24.167720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:24.167752 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:24.170084 (+ 2332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:24.170678 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:24.170682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:24.170684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:24.170810 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:24.171096 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:24.171099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:24.171101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:24.171349 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:24.171531 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:24.171705 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 20:59:24.171849 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 20:59:24.173801 1066 tablet_service.cc:2908] Rejecting scan request for tablet d4710222a5a345f7990f1682699974f9: Uninitialized: clean time has not yet been initialized
I20260929 20:59:24.176597 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 1023
I20260929 20:59:24.188232 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:24.177464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48698 (local address 127.0.97.1:40953)
0929 20:59:24.177650 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:24.177658 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:24.177990 (+ 332us) server_negotiation.cc:407] Connection header received
0929 20:59:24.178093 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:24.178096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:24.178155 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:24.178230 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:24.179273 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:24.179964 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:24.181122 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:24.181921 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:24.182014 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:24.183958 (+ 1944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:24.183963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:24.183966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:24.183991 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:24.186232 (+ 2241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:24.186760 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:24.186763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:24.186765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:24.186861 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:24.187131 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:24.187134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:24.187136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:24.187350 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:24.187507 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:24.187722 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 20:59:24.187879 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 20:59:24.188817 1146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:59:24.290135 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 1023
I20260929 20:59:24.307192 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 1157
Sep 29 20:59:24 dist-test-slave-1dms krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:59:24.321307 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:24.307927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39678 (local address 127.0.97.2:37511)
0929 20:59:24.308096 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:24.308102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:24.308150 (+ 48us) server_negotiation.cc:407] Connection header received
0929 20:59:24.308375 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:24.308379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:24.308445 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:24.308546 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:24.309841 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:24.310569 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:24.311960 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:24.312338 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:24.312460 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:24.316292 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:24.316305 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:24.316320 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:24.316350 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:24.318913 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:24.319491 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:24.319498 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:24.319500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:24.319607 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:24.319902 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:24.319905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:24.319907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:24.320337 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:24.320540 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:24.320771 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 20:59:24.320972 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:59:24.321950 1280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:59:24.408021 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 1157
I20260929 20:59:24.424522 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 1290
Sep 29 20:59:24 dist-test-slave-1dms krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:59:24.438781 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:24.425425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35832 (local address 127.0.97.3:34823)
0929 20:59:24.425662 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:24.425667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:24.425685 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:59:24.425750 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:24.425754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:24.425829 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:24.425913 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:24.427086 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:24.427845 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:24.429266 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:24.429642 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:24.429745 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:24.433611 (+ 3866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:24.433625 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:24.433641 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:24.433675 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:24.436172 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:24.436728 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:24.436735 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:24.436738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:24.436957 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:24.437343 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:24.437347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:24.437349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:24.437717 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:24.437922 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:24.438141 (+ 219us) server_negotiation.cc:299] Negotiation successful
0929 20:59:24.438297 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:59:24.439392 1413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:59:24.527423 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 1290
I20260929 20:59:24.543257 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 930
I20260929 20:59:24.544651 992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:59:24.648053 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 930
2026-09-29T20:59:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4169 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:59:24 dist-test-slave-1dms krb5kdc[1466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:59:24 dist-test-slave-1dms krb5kdc[1466](info): set up 2 sockets
Sep 29 20:59:24 dist-test-slave-1dms krb5kdc[1466](info): commencing operation
krb5kdc: starting...
W20260929 20:59:25.720808 388 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 20:59:25 dist-test-slave-1dms krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:59:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:59:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:59:25.890491 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37173
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:59:26.196172 1479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:26.196488 1479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:26.196659 1479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:26.203640 1479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:59:26.203768 1479 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 20:59:26.203860 1479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:26.203979 1479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:26.204061 1479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:59:26.204144 1479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:59:26.213644 1479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37173
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:26.214825 1479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:26.216089 1479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:26.227355 1484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:26.227409 1485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:26.227432 1487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:26.228390 1479 server_base.cc:1034] running on GCE node
I20260929 20:59:26.229187 1479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:26.230468 1479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:26.231703 1479 hybrid_clock.cc:648] HybridClock initialized: now 1790715566231646 us; error 37 us; skew 500 ppm
I20260929 20:59:26.232009 1479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:59:26 dist-test-slave-1dms krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:59:26.235080 1479 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:59:26.236450 1479 webserver.cc:460] Webserver started at http://127.0.97.62:37121/ using document root <none> and password file <none>
I20260929 20:59:26.237206 1479 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:26.237337 1479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:26.237655 1479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:26.240397 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "3658792fe6c64331aead6d6f2097bc00"
format_stamp: "Formatted at 2026-09-29 20:59:26 on dist-test-slave-1dms"
server_key: "789ddef5eb403ebc6e4598a8ceb694b0"
server_key_iv: "89aea36e04eb25a548c03a205aad84d0"
server_key_version: "encryptionkey@0"
I20260929 20:59:26.241215 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "3658792fe6c64331aead6d6f2097bc00"
format_stamp: "Formatted at 2026-09-29 20:59:26 on dist-test-slave-1dms"
server_key: "789ddef5eb403ebc6e4598a8ceb694b0"
server_key_iv: "89aea36e04eb25a548c03a205aad84d0"
server_key_version: "encryptionkey@0"
I20260929 20:59:26.246145 1479 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 20:59:26.249328 1494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:26.250532 1479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 20:59:26.250711 1479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "3658792fe6c64331aead6d6f2097bc00"
format_stamp: "Formatted at 2026-09-29 20:59:26 on dist-test-slave-1dms"
server_key: "789ddef5eb403ebc6e4598a8ceb694b0"
server_key_iv: "89aea36e04eb25a548c03a205aad84d0"
server_key_version: "encryptionkey@0"
I20260929 20:59:26.250918 1479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:26.269925 1479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:26.273731 1479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:26.274015 1479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:26.294999 1479 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37173
I20260929 20:59:26.295022 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37173 every 8 connection(s)
I20260929 20:59:26.296286 1479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:59:26.299933 1556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:26.305024 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 1479
I20260929 20:59:26.305255 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 20:59:26.305863 388 external_mini_cluster.cc:1428] Setting key 52b7f4dfc16a1496446fb282e49cbe9a
I20260929 20:59:26.311475 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3658792fe6c64331aead6d6f2097bc00: Bootstrap starting.
I20260929 20:59:26.315093 1556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3658792fe6c64331aead6d6f2097bc00: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:26.316155 1556 log.cc:824] T 00000000000000000000000000000000 P 3658792fe6c64331aead6d6f2097bc00: Log is configured to *not* fsync() on all Append() calls
I20260929 20:59:26.319142 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3658792fe6c64331aead6d6f2097bc00: No bootstrap required, opened a new log
Sep 29 20:59:26 dist-test-slave-1dms krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:59:26.328260 1556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3658792fe6c64331aead6d6f2097bc00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3658792fe6c64331aead6d6f2097bc00" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37173 } }
I20260929 20:59:26.328606 1556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3658792fe6c64331aead6d6f2097bc00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:26.328749 1556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3658792fe6c64331aead6d6f2097bc00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3658792fe6c64331aead6d6f2097bc00, State: Initialized, Role: FOLLOWER
I20260929 20:59:26.329428 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:26.307888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35290 (local address 127.0.97.62:37173)
0929 20:59:26.308738 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:26.308753 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:26.308816 (+ 63us) server_negotiation.cc:407] Connection header received
0929 20:59:26.311154 (+ 2338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:26.311181 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:26.311670 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:26.312100 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:26.313494 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:26.315425 (+ 1931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:26.316945 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:26.317810 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:26.317990 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:26.322205 (+ 4215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:26.322213 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:26.322231 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:26.322270 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:26.325269 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:26.326076 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:26.326081 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:26.326087 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:26.326255 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:26.326676 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:26.326679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:26.326681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:26.327177 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:26.327393 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:26.327825 (+ 432us) server_negotiation.cc:299] Negotiation successful
0929 20:59:26.328191 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":359,"threads_started":1}
I20260929 20:59:26.329334 1556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3658792fe6c64331aead6d6f2097bc00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3658792fe6c64331aead6d6f2097bc00" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37173 } }
I20260929 20:59:26.329624 1556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3658792fe6c64331aead6d6f2097bc00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:59:26.329795 1556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3658792fe6c64331aead6d6f2097bc00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:59:26.329975 1556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3658792fe6c64331aead6d6f2097bc00 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:26.332267 1556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3658792fe6c64331aead6d6f2097bc00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3658792fe6c64331aead6d6f2097bc00" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37173 } }
I20260929 20:59:26.332736 1556 leader_election.cc:302] T 00000000000000000000000000000000 P 3658792fe6c64331aead6d6f2097bc00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3658792fe6c64331aead6d6f2097bc00; no voters:
I20260929 20:59:26.333273 1556 leader_election.cc:288] T 00000000000000000000000000000000 P 3658792fe6c64331aead6d6f2097bc00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:59:26.333417 1561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3658792fe6c64331aead6d6f2097bc00 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:59:26.334138 1561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3658792fe6c64331aead6d6f2097bc00 [term 1 LEADER]: Becoming Leader. State: Replica: 3658792fe6c64331aead6d6f2097bc00, State: Running, Role: LEADER
I20260929 20:59:26.334750 1561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3658792fe6c64331aead6d6f2097bc00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3658792fe6c64331aead6d6f2097bc00" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37173 } }
I20260929 20:59:26.335304 1556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3658792fe6c64331aead6d6f2097bc00 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:59:26.337312 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3658792fe6c64331aead6d6f2097bc00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3658792fe6c64331aead6d6f2097bc00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3658792fe6c64331aead6d6f2097bc00" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37173 } } }
I20260929 20:59:26.337621 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3658792fe6c64331aead6d6f2097bc00 [sys.catalog]: This master's current role is: LEADER
I20260929 20:59:26.338435 1567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:59:26.338315 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3658792fe6c64331aead6d6f2097bc00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3658792fe6c64331aead6d6f2097bc00. Latest consensus state: current_term: 1 leader_uuid: "3658792fe6c64331aead6d6f2097bc00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3658792fe6c64331aead6d6f2097bc00" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37173 } } }
I20260929 20:59:26.338611 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3658792fe6c64331aead6d6f2097bc00 [sys.catalog]: This master's current role is: LEADER
I20260929 20:59:26.344692 1567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:59:26.353436 1567 catalog_manager.cc:1347] Generated new cluster ID: 132a9f1dd9284b7184ec415347776cc8
I20260929 20:59:26.353554 1567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:59:26.367357 1567 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:59:26.368607 1567 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:59:26.376951 1567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3658792fe6c64331aead6d6f2097bc00: Generated new TSK 0
I20260929 20:59:26.377969 1567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:59:26.424429 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37173
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:59:26.740968 1582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:26.741285 1582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:26.741451 1582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:26.748404 1582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:26.748545 1582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:26.748697 1582 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:59:26.748883 1582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:59:26.758497 1582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37173
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:26.759783 1582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:26.761108 1582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:26.774048 1587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:26.774101 1588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:26.774232 1590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:26.774621 1582 server_base.cc:1034] running on GCE node
I20260929 20:59:26.775285 1582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:26.776192 1582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:26.777433 1582 hybrid_clock.cc:648] HybridClock initialized: now 1790715566777370 us; error 39 us; skew 500 ppm
I20260929 20:59:26.777714 1582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:59:26 dist-test-slave-1dms krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:59:26.780597 1582 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:59:26.782032 1582 webserver.cc:460] Webserver started at http://127.0.97.1:41055/ using document root <none> and password file <none>
I20260929 20:59:26.782688 1582 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:26.782823 1582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:26.783133 1582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:26.785972 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-0/data/instance:
uuid: "74d4625e810948d996703bbe0ff7c98d"
format_stamp: "Formatted at 2026-09-29 20:59:26 on dist-test-slave-1dms"
server_key: "5f378eb0e72ea8ec262c2224499e267b"
server_key_iv: "824a15c77eab0edf41a059f85db794ef"
server_key_version: "encryptionkey@0"
I20260929 20:59:26.786710 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-0/wal/instance:
uuid: "74d4625e810948d996703bbe0ff7c98d"
format_stamp: "Formatted at 2026-09-29 20:59:26 on dist-test-slave-1dms"
server_key: "5f378eb0e72ea8ec262c2224499e267b"
server_key_iv: "824a15c77eab0edf41a059f85db794ef"
server_key_version: "encryptionkey@0"
I20260929 20:59:26.791591 1582 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:59:26.795022 1597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:26.796300 1582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:59:26.796468 1582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-0/wal
uuid: "74d4625e810948d996703bbe0ff7c98d"
format_stamp: "Formatted at 2026-09-29 20:59:26 on dist-test-slave-1dms"
server_key: "5f378eb0e72ea8ec262c2224499e267b"
server_key_iv: "824a15c77eab0edf41a059f85db794ef"
server_key_version: "encryptionkey@0"
I20260929 20:59:26.796679 1582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:26.817250 1582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:26.821177 1582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:26.821455 1582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:26.824090 1582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:59:26.824187 1582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:26.824316 1582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:59:26.824404 1582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:59:26 dist-test-slave-1dms krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:59:26.845949 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:26.824625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41339 (local address 127.0.97.62:37173)
0929 20:59:26.824824 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:26.824853 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:26.826324 (+ 1471us) server_negotiation.cc:407] Connection header received
0929 20:59:26.828360 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:26.828366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:26.828457 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:26.828571 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:26.830735 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:26.831764 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:26.833634 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:26.834234 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:26.834368 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:26.839195 (+ 4827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:26.839206 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:26.839210 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:26.839243 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:26.842968 (+ 3725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:26.843659 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:26.843665 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:26.843669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:26.843790 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:26.844205 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:26.844210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:26.844221 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:26.844482 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:26.844629 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:26.845380 (+ 751us) server_negotiation.cc:299] Negotiation successful
0929 20:59:26.845615 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 20:59:26.847371 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:26.825326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37173 (local address 127.0.97.1:41339)
0929 20:59:26.826172 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:26.826207 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:26.828065 (+ 1858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:26.828801 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:26.828809 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:26.829657 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:26.830534 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:26.830560 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:26.831906 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:26.831911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:26.833463 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:26.833477 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:26.834477 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:26.834485 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:26.834800 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:26.835578 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:59:26.835628 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:26.838977 (+ 3349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:26.843102 (+ 4125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:26.843110 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:26.843126 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:26.843533 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:26.843897 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:26.843903 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:26.843907 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:26.844090 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:26.844616 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:26.844623 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:26.845028 (+ 405us) client_negotiation.cc:769] Sending connection context
0929 20:59:26.845296 (+ 268us) client_negotiation.cc:240] Negotiation successful
0929 20:59:26.845839 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":433,"threads_started":1}
I20260929 20:59:26.868023 1663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:26.853083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37173 (local address 127.0.97.1:36549)
0929 20:59:26.853711 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:26.853727 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:26.853872 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:26.854192 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:26.854198 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:26.854717 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:26.854987 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:26.854995 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:26.856141 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:26.856146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:26.857845 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:26.857862 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:26.858735 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:26.858745 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:26.859017 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:26.859716 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:59:26.859747 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:26.861111 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:26.865125 (+ 4014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:26.865130 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:26.865138 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:26.865863 (+ 725us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:26.866319 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:26.866324 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:26.866328 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:26.866543 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:26.867180 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:26.867183 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:26.867333 (+ 150us) client_negotiation.cc:769] Sending connection context
0929 20:59:26.867460 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 20:59:26.867694 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":376,"threads_started":1}
I20260929 20:59:26.868243 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:26.852992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36549 (local address 127.0.97.62:37173)
0929 20:59:26.853150 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:26.853154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:26.853806 (+ 652us) server_negotiation.cc:407] Connection header received
0929 20:59:26.853983 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:26.853989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:26.854057 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:26.854156 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:26.855117 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:26.856012 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:26.858045 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:26.858514 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:26.858645 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:26.861292 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:26.861297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:26.861300 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:26.861328 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:26.864937 (+ 3609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:26.866022 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:26.866030 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:26.866036 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:26.866189 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:26.866685 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:26.866693 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:26.866699 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:26.867069 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:26.867215 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:26.867607 (+ 392us) server_negotiation.cc:299] Negotiation successful
0929 20:59:26.867838 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 20:59:26.879380 1582 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40541
I20260929 20:59:26.879467 1716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40541 every 8 connection(s)
I20260929 20:59:26.880573 1582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:59:26.881347 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 1582
I20260929 20:59:26.881556 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:59:26.882277 388 external_mini_cluster.cc:1428] Setting key 751da49acd0482c60c06080e63b40c51
I20260929 20:59:26.882799 1717 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37173
I20260929 20:59:26.883260 1717 heartbeater.cc:464] Registering TS with master...
I20260929 20:59:26.883927 1717 heartbeater.cc:511] Master 127.0.97.62:37173 requested a full tablet report, sending...
I20260929 20:59:26.885950 1511 ts_manager.cc:194] Registered new tserver with Master: 74d4625e810948d996703bbe0ff7c98d (127.0.97.1:40541)
I20260929 20:59:26.887343 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:41339
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:59:26.920984 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37173
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:59:27.227963 1721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:27.228225 1721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:27.228319 1721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:27.235474 1721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:27.235630 1721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:27.235734 1721 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:59:27.235813 1721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:59:27.245299 1721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37173
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:27.246474 1721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:27.247737 1721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:27.260919 1727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:27.261266 1726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:27.261668 1729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:27.261729 1721 server_base.cc:1034] running on GCE node
I20260929 20:59:27.262418 1721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:27.263375 1721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:27.264648 1721 hybrid_clock.cc:648] HybridClock initialized: now 1790715567264580 us; error 68 us; skew 500 ppm
I20260929 20:59:27.265027 1721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:59:27 dist-test-slave-1dms krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:59:27.268115 1721 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:59:27.269501 1721 webserver.cc:460] Webserver started at http://127.0.97.2:35667/ using document root <none> and password file <none>
I20260929 20:59:27.270187 1721 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:27.270320 1721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:27.270671 1721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:27.273591 1721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-1/data/instance:
uuid: "47d0abd5486b4bfdbfb7059499ff1d34"
format_stamp: "Formatted at 2026-09-29 20:59:27 on dist-test-slave-1dms"
server_key: "deaa94afd7cb29e9f8cf1b754493e39b"
server_key_iv: "f1e37166d1fc0c920450738e004deec4"
server_key_version: "encryptionkey@0"
I20260929 20:59:27.274348 1721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-1/wal/instance:
uuid: "47d0abd5486b4bfdbfb7059499ff1d34"
format_stamp: "Formatted at 2026-09-29 20:59:27 on dist-test-slave-1dms"
server_key: "deaa94afd7cb29e9f8cf1b754493e39b"
server_key_iv: "f1e37166d1fc0c920450738e004deec4"
server_key_version: "encryptionkey@0"
I20260929 20:59:27.279335 1721 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:59:27.282939 1736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:27.284265 1721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:59:27.284451 1721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-1/wal
uuid: "47d0abd5486b4bfdbfb7059499ff1d34"
format_stamp: "Formatted at 2026-09-29 20:59:27 on dist-test-slave-1dms"
server_key: "deaa94afd7cb29e9f8cf1b754493e39b"
server_key_iv: "f1e37166d1fc0c920450738e004deec4"
server_key_version: "encryptionkey@0"
I20260929 20:59:27.284677 1721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:27.303182 1721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:27.307497 1721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:27.307767 1721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:27.310412 1721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:59:27.310515 1721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:27.310654 1721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:59:27.310731 1721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:59:27 dist-test-slave-1dms krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:59:27.334872 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:27.311144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45121 (local address 127.0.97.62:37173)
0929 20:59:27.311371 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:27.311379 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:27.312753 (+ 1374us) server_negotiation.cc:407] Connection header received
0929 20:59:27.314959 (+ 2206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:27.314965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:27.315064 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:27.315196 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:27.317526 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:27.318541 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:27.321257 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:27.321855 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:27.321993 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:27.327060 (+ 5067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:27.327066 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:27.327069 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:27.327102 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:27.330891 (+ 3789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:27.331860 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:27.331869 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:27.331876 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:27.332038 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:27.332893 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:27.332898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:27.332901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:27.333292 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:27.333435 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:27.334305 (+ 870us) server_negotiation.cc:299] Negotiation successful
0929 20:59:27.334529 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 20:59:27.336671 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:27.311751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37173 (local address 127.0.97.2:45121)
0929 20:59:27.312583 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:27.312624 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:27.314624 (+ 2000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:27.315677 (+ 1053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:27.315685 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:27.316470 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:27.317353 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:27.317374 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:27.318696 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:27.318701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:27.321100 (+ 2399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:27.321113 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:27.322128 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:27.322137 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:27.322339 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:27.323177 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:59:27.323200 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:27.326732 (+ 3532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:27.331043 (+ 4311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:27.331053 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:27.331076 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:27.331693 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:27.332203 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:27.332213 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:27.332218 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:27.332732 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:27.333411 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:27.333423 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:27.333928 (+ 505us) client_negotiation.cc:769] Sending connection context
0929 20:59:27.334206 (+ 278us) client_negotiation.cc:240] Negotiation successful
0929 20:59:27.334781 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":385,"threads_started":1}
I20260929 20:59:27.357856 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:27.342533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60809 (local address 127.0.97.62:37173)
0929 20:59:27.342687 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:27.342691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:27.343348 (+ 657us) server_negotiation.cc:407] Connection header received
0929 20:59:27.343543 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:27.343554 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:27.343611 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:27.343709 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:27.344998 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:27.346486 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:27.347882 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:27.348784 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:27.348966 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:27.351380 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:27.351385 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:27.351388 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:27.351418 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:27.354491 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:27.355515 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:27.355522 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:27.355527 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:27.355683 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:27.356137 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:27.356146 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:27.356152 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:27.356632 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:27.356796 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:27.357281 (+ 485us) server_negotiation.cc:299] Negotiation successful
0929 20:59:27.357493 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 20:59:27.357949 1821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:27.342649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37173 (local address 127.0.97.2:60809)
0929 20:59:27.343248 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:27.343263 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:27.343418 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:27.343749 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:27.343753 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:27.344344 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:27.344808 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:27.344828 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:27.346628 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:27.346632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:27.347745 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:27.347757 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:27.348988 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:27.348993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:27.349113 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:27.349794 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:59:27.349821 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:27.351206 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:27.354618 (+ 3412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:27.354626 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:27.354633 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:27.355380 (+ 747us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:27.355837 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:27.355842 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:27.355846 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:27.356007 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:27.356812 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:27.356823 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:27.357022 (+ 199us) client_negotiation.cc:769] Sending connection context
0929 20:59:27.357184 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 20:59:27.357493 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":339,"threads_started":1}
I20260929 20:59:27.363647 1721 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35735
I20260929 20:59:27.363667 1855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35735 every 8 connection(s)
I20260929 20:59:27.364964 1721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:59:27.366662 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 1721
I20260929 20:59:27.366873 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:59:27.367313 388 external_mini_cluster.cc:1428] Setting key f480be85fde103c3d2e5315f6eb9c9b1
I20260929 20:59:27.367229 1856 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37173
I20260929 20:59:27.368156 1856 heartbeater.cc:464] Registering TS with master...
I20260929 20:59:27.368875 1856 heartbeater.cc:511] Master 127.0.97.62:37173 requested a full tablet report, sending...
I20260929 20:59:27.370198 1511 ts_manager.cc:194] Registered new tserver with Master: 47d0abd5486b4bfdbfb7059499ff1d34 (127.0.97.2:35735)
I20260929 20:59:27.371441 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45121
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:59:27.407639 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37173
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:59:27.721776 1860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:27.722043 1860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:27.722149 1860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:27.730340 1860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:27.730486 1860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:27.730646 1860 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:59:27.730789 1860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:59:27.741133 1860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37173
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:27.742548 1860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:27.744040 1860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:27.757506 1866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:27.757499 1865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:27.757685 1868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:27.758368 1860 server_base.cc:1034] running on GCE node
I20260929 20:59:27.758971 1860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:27.759851 1860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:27.761055 1860 hybrid_clock.cc:648] HybridClock initialized: now 1790715567760990 us; error 38 us; skew 500 ppm
I20260929 20:59:27.761361 1860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:59:27 dist-test-slave-1dms krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:59:27.764535 1860 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:59:27.766124 1860 webserver.cc:460] Webserver started at http://127.0.97.3:38419/ using document root <none> and password file <none>
I20260929 20:59:27.766820 1860 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:27.766956 1860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:27.767285 1860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:27.770869 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-2/data/instance:
uuid: "5f1413b4f1864fa99efd5f319ff7d60b"
format_stamp: "Formatted at 2026-09-29 20:59:27 on dist-test-slave-1dms"
server_key: "bb94488b139efdf3ee1811a787218f15"
server_key_iv: "3b44cb6460e32aec9f8ed30606c06bc8"
server_key_version: "encryptionkey@0"
I20260929 20:59:27.771677 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-2/wal/instance:
uuid: "5f1413b4f1864fa99efd5f319ff7d60b"
format_stamp: "Formatted at 2026-09-29 20:59:27 on dist-test-slave-1dms"
server_key: "bb94488b139efdf3ee1811a787218f15"
server_key_iv: "3b44cb6460e32aec9f8ed30606c06bc8"
server_key_version: "encryptionkey@0"
I20260929 20:59:27.777344 1860 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 20:59:27.780925 1875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:27.782235 1860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:59:27.782389 1860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-2/wal
uuid: "5f1413b4f1864fa99efd5f319ff7d60b"
format_stamp: "Formatted at 2026-09-29 20:59:27 on dist-test-slave-1dms"
server_key: "bb94488b139efdf3ee1811a787218f15"
server_key_iv: "3b44cb6460e32aec9f8ed30606c06bc8"
server_key_version: "encryptionkey@0"
I20260929 20:59:27.782608 1860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:27.811614 1860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:27.816069 1860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:27.816355 1860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:27.819017 1860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:59:27.819103 1860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:27.819344 1860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:59:27.819425 1860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:59:27 dist-test-slave-1dms krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:59:27.840758 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:27.819370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39731 (local address 127.0.97.62:37173)
0929 20:59:27.819545 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:27.819551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:27.820935 (+ 1384us) server_negotiation.cc:407] Connection header received
0929 20:59:27.823002 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:27.823007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:27.823094 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:27.823198 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:27.824744 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:27.826138 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:27.828621 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:27.829263 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:27.829422 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:27.833807 (+ 4385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:27.833812 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:27.833816 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:27.833845 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:27.837051 (+ 3206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:27.838187 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:27.838193 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:27.838198 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:27.838357 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:27.838934 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:27.838937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:27.838939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:27.839179 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:27.839280 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:27.840279 (+ 999us) server_negotiation.cc:299] Negotiation successful
0929 20:59:27.840470 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:59:27.842729 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:27.819915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37173 (local address 127.0.97.3:39731)
0929 20:59:27.820744 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:27.820812 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:27.822697 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:27.823417 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:27.823425 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:27.823908 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:27.824473 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:27.824489 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:27.826320 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:27.826332 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:27.828482 (+ 2150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:27.828493 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:27.829709 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:27.829714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:27.829895 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:27.830639 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:59:27.830669 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:27.833591 (+ 2922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:27.837192 (+ 3601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:27.837206 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:27.837229 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:27.838037 (+ 808us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:27.838495 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:27.838503 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:27.838509 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:27.838821 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:27.839321 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:27.839332 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:27.839864 (+ 532us) client_negotiation.cc:769] Sending connection context
0929 20:59:27.840230 (+ 366us) client_negotiation.cc:240] Negotiation successful
0929 20:59:27.840905 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":385,"threads_started":1}
I20260929 20:59:27.863407 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:27.848374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42485 (local address 127.0.97.62:37173)
0929 20:59:27.848522 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:27.848526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:27.849672 (+ 1146us) server_negotiation.cc:407] Connection header received
0929 20:59:27.849876 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:27.849882 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:27.849947 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:27.850043 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:27.851408 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:27.852451 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:27.854301 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:27.854903 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:27.855025 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:27.857408 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:27.857414 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:27.857419 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:27.857459 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:27.860677 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:27.861478 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:27.861483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:27.861487 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:27.861630 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:27.861994 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:27.861999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:27.862002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:27.862257 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:27.862414 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:27.862812 (+ 398us) server_negotiation.cc:299] Negotiation successful
0929 20:59:27.863012 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 20:59:27.863495 1934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:27.848501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37173 (local address 127.0.97.3:42485)
0929 20:59:27.849579 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:27.849595 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:27.849748 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:27.850061 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:27.850065 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:27.850743 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:27.851277 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:27.851288 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:27.852575 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:27.852581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:27.854102 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:27.854120 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:27.855084 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:27.855091 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:27.855348 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:27.856048 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:59:27.856067 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:27.857236 (+ 1169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:27.860799 (+ 3563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:27.860804 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:27.860809 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:27.861356 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:27.861735 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:27.861741 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:27.861745 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:27.861884 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:27.862370 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:27.862376 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:27.862549 (+ 173us) client_negotiation.cc:769] Sending connection context
0929 20:59:27.862703 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 20:59:27.863051 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"thread_start_us":460,"threads_started":1}
I20260929 20:59:27.875366 1860 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41889
I20260929 20:59:27.875392 1994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41889 every 8 connection(s)
I20260929 20:59:27.876549 1860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:59:27.878814 1995 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37173
I20260929 20:59:27.879065 1995 heartbeater.cc:464] Registering TS with master...
I20260929 20:59:27.879807 1995 heartbeater.cc:511] Master 127.0.97.62:37173 requested a full tablet report, sending...
I20260929 20:59:27.881321 1511 ts_manager.cc:194] Registered new tserver with Master: 5f1413b4f1864fa99efd5f319ff7d60b (127.0.97.3:41889)
I20260929 20:59:27.882134 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39731
I20260929 20:59:27.883632 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 1860
I20260929 20:59:27.883819 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:59:27.884358 388 external_mini_cluster.cc:1428] Setting key 91be62a139b4d7d9c4323b8dad0ba53f
I20260929 20:59:27.887118 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:59:27.889850 1717 heartbeater.cc:503] Master 127.0.97.62:37173 was elected leader, sending a full tablet report...
Sep 29 20:59:27 dist-test-slave-1dms krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715567, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260929 20:59:27.991506 388 mini_kdc.cc:317] Time spent kinit for test-user: real 0.103s user 0.004s sys 0.006s
Sep 29 20:59:27 dist-test-slave-1dms krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715567, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:59:28.008807 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:27.995241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35296 (local address 127.0.97.62:37173)
0929 20:59:27.995473 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:27.995481 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:27.995662 (+ 181us) server_negotiation.cc:407] Connection header received
0929 20:59:27.995896 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:27.995901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:27.995976 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:27.996082 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:27.997604 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:27.998320 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:27.999648 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:28.000110 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:28.000248 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:28.003925 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:28.003930 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:28.003933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:28.003967 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:28.006379 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:28.006980 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:28.006991 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:28.006995 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:28.007098 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:28.007465 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:28.007473 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:28.007476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:28.007720 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:28.007965 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:28.008280 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 20:59:28.008487 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 20:59:28.012773 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:59:28.019704 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:59:28.051020 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:28.031403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33254 (local address 127.0.97.3:41889)
0929 20:59:28.033264 (+ 1861us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:28.033278 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:28.041305 (+ 8027us) server_negotiation.cc:407] Connection header received
0929 20:59:28.042445 (+ 1140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:28.042458 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:28.042804 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:28.043002 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:59:28.044226 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:28.045901 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:28.048305 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:28.049598 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:28.049717 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:28.049986 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:28.050535 (+ 549us) server_negotiation.cc:299] Negotiation successful
0929 20:59:28.050691 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1638,"thread_start_us":1463,"threads_started":1}
I20260929 20:59:28.051033 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:28.031235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41889 (local address 127.0.0.1:33254)
0929 20:59:28.041126 (+ 9891us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:28.041165 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:28.041971 (+ 806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:28.043475 (+ 1504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:28.043484 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:28.043520 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:59:28.044000 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:28.044012 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:28.046029 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:28.046037 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:28.048148 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:28.048158 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:28.050105 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:28.050109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:28.050314 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:28.050347 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 20:59:28.050488 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 20:59:28.050663 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9652,"thread_start_us":12985,"threads_started":1}
I20260929 20:59:28.051828 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:28.033543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35735 (local address 127.0.0.1:56834)
0929 20:59:28.044085 (+ 10542us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:28.044103 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:28.044209 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:28.045081 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:28.045085 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:28.045107 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:59:28.045524 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:28.045536 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:28.046949 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:28.046955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:28.049034 (+ 2079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:28.049047 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:28.051022 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:28.051029 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:28.051295 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:28.051324 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 20:59:28.051394 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 20:59:28.051509 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10413,"thread_start_us":7208,"threads_started":1}
I20260929 20:59:28.051991 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:28.033847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56834 (local address 127.0.97.2:35735)
0929 20:59:28.044170 (+ 10323us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:28.044180 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:28.044209 (+ 29us) server_negotiation.cc:407] Connection header received
0929 20:59:28.044393 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:28.044401 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:28.044636 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:28.044895 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:59:28.045753 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:28.046817 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:28.049205 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:28.050565 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:28.050688 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:28.050874 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:28.051589 (+ 715us) server_negotiation.cc:299] Negotiation successful
0929 20:59:28.051737 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10149,"thread_start_us":10030,"threads_started":1}
I20260929 20:59:28.053158 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:28.031432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40541 (local address 127.0.0.1:51220)
0929 20:59:28.044464 (+ 13032us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:28.044481 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:28.044628 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:28.045435 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:28.045439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:28.045472 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:59:28.045933 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:28.045944 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:28.047703 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:28.047708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:28.049872 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:28.049887 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:28.052376 (+ 2489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:28.052382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:28.052543 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:28.052573 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 20:59:28.052690 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 20:59:28.052854 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12163,"thread_start_us":11980,"threads_started":1}
I20260929 20:59:28.053200 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:28.031780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51220 (local address 127.0.97.1:40541)
0929 20:59:28.043233 (+ 11453us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:28.043245 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:28.044569 (+ 1324us) server_negotiation.cc:407] Connection header received
0929 20:59:28.044802 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:28.044812 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:28.045035 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:28.045213 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:59:28.046099 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:28.047571 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:28.050042 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:28.051835 (+ 1793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:28.052331 (+ 496us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:28.052520 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:28.052743 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 20:59:28.052917 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11233,"thread_start_us":12537,"threads_started":1}
I20260929 20:59:28.055044 1786 tablet_service.cc:1467] Processing CreateTablet for tablet 9bc05823db9f4760972eb5863b29de94 (DEFAULT_TABLE table=test-table [id=1d7be766919f40b5af7776894a222a2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:59:28.055045 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 9bc05823db9f4760972eb5863b29de94 (DEFAULT_TABLE table=test-table [id=1d7be766919f40b5af7776894a222a2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:59:28.056443 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bc05823db9f4760972eb5863b29de94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:28.056444 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bc05823db9f4760972eb5863b29de94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:28.058735 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 9bc05823db9f4760972eb5863b29de94 (DEFAULT_TABLE table=test-table [id=1d7be766919f40b5af7776894a222a2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:59:28.059821 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bc05823db9f4760972eb5863b29de94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:28.071288 2015 tablet_bootstrap.cc:492] T 9bc05823db9f4760972eb5863b29de94 P 5f1413b4f1864fa99efd5f319ff7d60b: Bootstrap starting.
I20260929 20:59:28.073479 2016 tablet_bootstrap.cc:492] T 9bc05823db9f4760972eb5863b29de94 P 47d0abd5486b4bfdbfb7059499ff1d34: Bootstrap starting.
I20260929 20:59:28.075328 2014 tablet_bootstrap.cc:492] T 9bc05823db9f4760972eb5863b29de94 P 74d4625e810948d996703bbe0ff7c98d: Bootstrap starting.
I20260929 20:59:28.076534 2015 tablet_bootstrap.cc:654] T 9bc05823db9f4760972eb5863b29de94 P 5f1413b4f1864fa99efd5f319ff7d60b: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:28.077234 2016 tablet_bootstrap.cc:654] T 9bc05823db9f4760972eb5863b29de94 P 47d0abd5486b4bfdbfb7059499ff1d34: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:28.077678 2015 log.cc:824] T 9bc05823db9f4760972eb5863b29de94 P 5f1413b4f1864fa99efd5f319ff7d60b: Log is configured to *not* fsync() on all Append() calls
I20260929 20:59:28.078390 2016 log.cc:824] T 9bc05823db9f4760972eb5863b29de94 P 47d0abd5486b4bfdbfb7059499ff1d34: Log is configured to *not* fsync() on all Append() calls
I20260929 20:59:28.078596 2014 tablet_bootstrap.cc:654] T 9bc05823db9f4760972eb5863b29de94 P 74d4625e810948d996703bbe0ff7c98d: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:28.079600 2014 log.cc:824] T 9bc05823db9f4760972eb5863b29de94 P 74d4625e810948d996703bbe0ff7c98d: Log is configured to *not* fsync() on all Append() calls
I20260929 20:59:28.080909 2015 tablet_bootstrap.cc:492] T 9bc05823db9f4760972eb5863b29de94 P 5f1413b4f1864fa99efd5f319ff7d60b: No bootstrap required, opened a new log
I20260929 20:59:28.081099 2016 tablet_bootstrap.cc:492] T 9bc05823db9f4760972eb5863b29de94 P 47d0abd5486b4bfdbfb7059499ff1d34: No bootstrap required, opened a new log
I20260929 20:59:28.081224 2015 ts_tablet_manager.cc:1397] T 9bc05823db9f4760972eb5863b29de94 P 5f1413b4f1864fa99efd5f319ff7d60b: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20260929 20:59:28.081766 2016 ts_tablet_manager.cc:1397] T 9bc05823db9f4760972eb5863b29de94 P 47d0abd5486b4bfdbfb7059499ff1d34: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.005s
I20260929 20:59:28.082748 2014 tablet_bootstrap.cc:492] T 9bc05823db9f4760972eb5863b29de94 P 74d4625e810948d996703bbe0ff7c98d: No bootstrap required, opened a new log
I20260929 20:59:28.082988 2014 ts_tablet_manager.cc:1397] T 9bc05823db9f4760972eb5863b29de94 P 74d4625e810948d996703bbe0ff7c98d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 20:59:28.090517 2015 raft_consensus.cc:348] T 9bc05823db9f4760972eb5863b29de94 P 5f1413b4f1864fa99efd5f319ff7d60b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } } peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } }
I20260929 20:59:28.090888 2015 raft_consensus.cc:374] T 9bc05823db9f4760972eb5863b29de94 P 5f1413b4f1864fa99efd5f319ff7d60b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:28.090984 2015 raft_consensus.cc:729] T 9bc05823db9f4760972eb5863b29de94 P 5f1413b4f1864fa99efd5f319ff7d60b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f1413b4f1864fa99efd5f319ff7d60b, State: Initialized, Role: FOLLOWER
I20260929 20:59:28.091656 2015 consensus_queue.cc:260] T 9bc05823db9f4760972eb5863b29de94 P 5f1413b4f1864fa99efd5f319ff7d60b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } } peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } }
I20260929 20:59:28.092124 2016 raft_consensus.cc:348] T 9bc05823db9f4760972eb5863b29de94 P 47d0abd5486b4bfdbfb7059499ff1d34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } } peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } }
I20260929 20:59:28.092484 2016 raft_consensus.cc:374] T 9bc05823db9f4760972eb5863b29de94 P 47d0abd5486b4bfdbfb7059499ff1d34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:28.092571 2016 raft_consensus.cc:729] T 9bc05823db9f4760972eb5863b29de94 P 47d0abd5486b4bfdbfb7059499ff1d34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47d0abd5486b4bfdbfb7059499ff1d34, State: Initialized, Role: FOLLOWER
I20260929 20:59:28.093163 2016 consensus_queue.cc:260] T 9bc05823db9f4760972eb5863b29de94 P 47d0abd5486b4bfdbfb7059499ff1d34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } } peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } }
I20260929 20:59:28.093524 2015 ts_tablet_manager.cc:1428] T 9bc05823db9f4760972eb5863b29de94 P 5f1413b4f1864fa99efd5f319ff7d60b: Time spent starting tablet: real 0.012s user 0.006s sys 0.005s
I20260929 20:59:28.093981 1995 heartbeater.cc:503] Master 127.0.97.62:37173 was elected leader, sending a full tablet report...
I20260929 20:59:28.094617 2016 ts_tablet_manager.cc:1428] T 9bc05823db9f4760972eb5863b29de94 P 47d0abd5486b4bfdbfb7059499ff1d34: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260929 20:59:28.095829 1856 heartbeater.cc:503] Master 127.0.97.62:37173 was elected leader, sending a full tablet report...
I20260929 20:59:28.096724 2014 raft_consensus.cc:348] T 9bc05823db9f4760972eb5863b29de94 P 74d4625e810948d996703bbe0ff7c98d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } } peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } }
I20260929 20:59:28.098073 2014 raft_consensus.cc:374] T 9bc05823db9f4760972eb5863b29de94 P 74d4625e810948d996703bbe0ff7c98d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:28.098179 2014 raft_consensus.cc:729] T 9bc05823db9f4760972eb5863b29de94 P 74d4625e810948d996703bbe0ff7c98d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74d4625e810948d996703bbe0ff7c98d, State: Initialized, Role: FOLLOWER
I20260929 20:59:28.098750 2014 consensus_queue.cc:260] T 9bc05823db9f4760972eb5863b29de94 P 74d4625e810948d996703bbe0ff7c98d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } } peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } }
I20260929 20:59:28.100603 2014 ts_tablet_manager.cc:1428] T 9bc05823db9f4760972eb5863b29de94 P 74d4625e810948d996703bbe0ff7c98d: Time spent starting tablet: real 0.018s user 0.011s sys 0.004s
W20260929 20:59:28.117098 1857 tablet.cc:2369] T 9bc05823db9f4760972eb5863b29de94 P 47d0abd5486b4bfdbfb7059499ff1d34: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:59:28.128118 1996 tablet.cc:2369] T 9bc05823db9f4760972eb5863b29de94 P 5f1413b4f1864fa99efd5f319ff7d60b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:59:28.132975 1718 tablet.cc:2369] T 9bc05823db9f4760972eb5863b29de94 P 74d4625e810948d996703bbe0ff7c98d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:59:28.187564 2020 raft_consensus.cc:482] T 9bc05823db9f4760972eb5863b29de94 P 5f1413b4f1864fa99efd5f319ff7d60b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:59:28.187896 2020 raft_consensus.cc:504] T 9bc05823db9f4760972eb5863b29de94 P 5f1413b4f1864fa99efd5f319ff7d60b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } } peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } }
I20260929 20:59:28.190158 2020 leader_election.cc:288] T 9bc05823db9f4760972eb5863b29de94 P 5f1413b4f1864fa99efd5f319ff7d60b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74d4625e810948d996703bbe0ff7c98d (127.0.97.1:40541), 47d0abd5486b4bfdbfb7059499ff1d34 (127.0.97.2:35735)
I20260929 20:59:28.197146 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:28.190549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54591 (local address 127.0.97.1:40541)
0929 20:59:28.190805 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:28.190814 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:28.191486 (+ 672us) server_negotiation.cc:407] Connection header received
0929 20:59:28.191709 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:28.191716 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:28.191816 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:28.191912 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:59:28.192384 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:28.193342 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:28.195066 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:28.196073 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:28.196213 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:28.196328 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:28.196734 (+ 406us) server_negotiation.cc:299] Negotiation successful
0929 20:59:28.196879 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 20:59:28.197333 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:28.190481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40541 (local address 127.0.97.3:54591)
0929 20:59:28.191362 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:28.191377 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:28.191536 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:28.191921 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:28.191924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:28.191941 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:59:28.192235 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:28.192241 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:28.193475 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:28.193481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:28.194904 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:28.194916 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:28.196298 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:28.196308 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:28.196529 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:28.196583 (+ 54us) client_negotiation.cc:769] Sending connection context
0929 20:59:28.196681 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 20:59:28.196875 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"mutex_wait_us":22,"spinlock_wait_cycles":22400,"thread_start_us":588,"threads_started":1}
I20260929 20:59:28.197752 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:28.190430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35735 (local address 127.0.97.3:43145)
0929 20:59:28.190787 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:28.190826 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:28.190946 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:28.191366 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:28.191373 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:28.191404 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:59:28.191625 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:28.191638 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:28.192932 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:28.192940 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:28.195367 (+ 2427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:28.195385 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:28.196978 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:28.196984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:28.197201 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:28.197234 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 20:59:28.197334 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 20:59:28.197465 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":59}
I20260929 20:59:28.197911 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:28.190651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43145 (local address 127.0.97.2:35735)
0929 20:59:28.190882 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:28.190888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:28.190907 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:59:28.191138 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:28.191145 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:28.191228 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:28.191332 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:59:28.191791 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:28.192781 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:28.195529 (+ 2748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:28.196610 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:28.196761 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:28.196896 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:28.197534 (+ 638us) server_negotiation.cc:299] Negotiation successful
0929 20:59:28.197667 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 20:59:28.198350 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bc05823db9f4760972eb5863b29de94" candidate_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74d4625e810948d996703bbe0ff7c98d" is_pre_election: true
I20260929 20:59:28.198874 1672 raft_consensus.cc:2454] T 9bc05823db9f4760972eb5863b29de94 P 74d4625e810948d996703bbe0ff7c98d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f1413b4f1864fa99efd5f319ff7d60b in term 0.
I20260929 20:59:28.199486 1876 leader_election.cc:302] T 9bc05823db9f4760972eb5863b29de94 P 5f1413b4f1864fa99efd5f319ff7d60b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f1413b4f1864fa99efd5f319ff7d60b, 74d4625e810948d996703bbe0ff7c98d; no voters:
I20260929 20:59:28.199577 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bc05823db9f4760972eb5863b29de94" candidate_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" is_pre_election: true
I20260929 20:59:28.199923 2020 raft_consensus.cc:2781] T 9bc05823db9f4760972eb5863b29de94 P 5f1413b4f1864fa99efd5f319ff7d60b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:59:28.200062 2020 raft_consensus.cc:482] T 9bc05823db9f4760972eb5863b29de94 P 5f1413b4f1864fa99efd5f319ff7d60b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:59:28.200067 1806 raft_consensus.cc:2454] T 9bc05823db9f4760972eb5863b29de94 P 47d0abd5486b4bfdbfb7059499ff1d34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f1413b4f1864fa99efd5f319ff7d60b in term 0.
I20260929 20:59:28.200177 2020 raft_consensus.cc:3037] T 9bc05823db9f4760972eb5863b29de94 P 5f1413b4f1864fa99efd5f319ff7d60b [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:28.282132 2020 raft_consensus.cc:504] T 9bc05823db9f4760972eb5863b29de94 P 5f1413b4f1864fa99efd5f319ff7d60b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } } peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } }
I20260929 20:59:28.283044 2020 leader_election.cc:288] T 9bc05823db9f4760972eb5863b29de94 P 5f1413b4f1864fa99efd5f319ff7d60b [CANDIDATE]: Term 1 election: Requested vote from peers 74d4625e810948d996703bbe0ff7c98d (127.0.97.1:40541), 47d0abd5486b4bfdbfb7059499ff1d34 (127.0.97.2:35735)
I20260929 20:59:28.283653 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bc05823db9f4760972eb5863b29de94" candidate_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47d0abd5486b4bfdbfb7059499ff1d34"
I20260929 20:59:28.283671 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bc05823db9f4760972eb5863b29de94" candidate_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74d4625e810948d996703bbe0ff7c98d"
I20260929 20:59:28.283885 1806 raft_consensus.cc:3037] T 9bc05823db9f4760972eb5863b29de94 P 47d0abd5486b4bfdbfb7059499ff1d34 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:28.283898 1672 raft_consensus.cc:3037] T 9bc05823db9f4760972eb5863b29de94 P 74d4625e810948d996703bbe0ff7c98d [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:28.365759 1806 raft_consensus.cc:2454] T 9bc05823db9f4760972eb5863b29de94 P 47d0abd5486b4bfdbfb7059499ff1d34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f1413b4f1864fa99efd5f319ff7d60b in term 1.
I20260929 20:59:28.365780 1672 raft_consensus.cc:2454] T 9bc05823db9f4760972eb5863b29de94 P 74d4625e810948d996703bbe0ff7c98d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f1413b4f1864fa99efd5f319ff7d60b in term 1.
I20260929 20:59:28.366340 1876 leader_election.cc:302] T 9bc05823db9f4760972eb5863b29de94 P 5f1413b4f1864fa99efd5f319ff7d60b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f1413b4f1864fa99efd5f319ff7d60b, 74d4625e810948d996703bbe0ff7c98d; no voters:
I20260929 20:59:28.367079 2020 raft_consensus.cc:2781] T 9bc05823db9f4760972eb5863b29de94 P 5f1413b4f1864fa99efd5f319ff7d60b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:59:28.368147 2020 raft_consensus.cc:686] T 9bc05823db9f4760972eb5863b29de94 P 5f1413b4f1864fa99efd5f319ff7d60b [term 1 LEADER]: Becoming Leader. State: Replica: 5f1413b4f1864fa99efd5f319ff7d60b, State: Running, Role: LEADER
I20260929 20:59:28.369282 2020 consensus_queue.cc:237] T 9bc05823db9f4760972eb5863b29de94 P 5f1413b4f1864fa99efd5f319ff7d60b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } } peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } }
I20260929 20:59:28.374796 1510 catalog_manager.cc:5495] T 9bc05823db9f4760972eb5863b29de94 P 5f1413b4f1864fa99efd5f319ff7d60b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f1413b4f1864fa99efd5f319ff7d60b (127.0.97.3). New cstate: current_term: 1 leader_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } health_report { overall_health: UNKNOWN } } }
I20260929 20:59:28.417811 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:28.412677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43340 (local address 127.0.97.3:41889)
0929 20:59:28.412951 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:28.412956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:28.412975 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:59:28.413225 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:28.413229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:28.413313 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:28.413428 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:59:28.413728 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:28.414462 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:28.415685 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:28.416076 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:28.416174 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:28.416644 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:59:28.417211 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:59:28.417320 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:28.417436 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 20:59:28.417507 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 20:59:28.426301 1672 raft_consensus.cc:1261] T 9bc05823db9f4760972eb5863b29de94 P 74d4625e810948d996703bbe0ff7c98d [term 1 FOLLOWER]: Refusing update from remote peer 5f1413b4f1864fa99efd5f319ff7d60b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:59:28.426301 1806 raft_consensus.cc:1261] T 9bc05823db9f4760972eb5863b29de94 P 47d0abd5486b4bfdbfb7059499ff1d34 [term 1 FOLLOWER]: Refusing update from remote peer 5f1413b4f1864fa99efd5f319ff7d60b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:59:28.427349 2024 consensus_queue.cc:1035] T 9bc05823db9f4760972eb5863b29de94 P 5f1413b4f1864fa99efd5f319ff7d60b [LEADER]: Connected to new peer: Peer: permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:59:28.427680 2020 consensus_queue.cc:1035] T 9bc05823db9f4760972eb5863b29de94 P 5f1413b4f1864fa99efd5f319ff7d60b [LEADER]: Connected to new peer: Peer: permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:59:28.440172 2028 mvcc.cc:205] Tried to move back new op lower bound from 7334770968264093696 to 7334770968046120960. Current Snapshot: MvccSnapshot[applied={T|T < 7334770968264093696}]
I20260929 20:59:28.457016 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:59:28.457302 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:59:28.461143 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3658792fe6c64331aead6d6f2097bc00: Sending DeleteTablet for 3 replicas of tablet 9bc05823db9f4760972eb5863b29de94
I20260929 20:59:28.462203 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bc05823db9f4760972eb5863b29de94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:59:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33254
I20260929 20:59:28.462808 1786 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bc05823db9f4760972eb5863b29de94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:59:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56834
I20260929 20:59:28.465603 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bc05823db9f4760972eb5863b29de94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:59:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51220
I20260929 20:59:28.468935 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:59:28.469790 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:59:28.474882 2036 tablet_replica.cc:330] T 9bc05823db9f4760972eb5863b29de94 P 5f1413b4f1864fa99efd5f319ff7d60b: stopping tablet replica
I20260929 20:59:28.475890 2036 raft_consensus.cc:2229] T 9bc05823db9f4760972eb5863b29de94 P 5f1413b4f1864fa99efd5f319ff7d60b [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:59:28.476629 2036 raft_consensus.cc:2258] T 9bc05823db9f4760972eb5863b29de94 P 5f1413b4f1864fa99efd5f319ff7d60b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:59:28.477308 2038 tablet_replica.cc:330] T 9bc05823db9f4760972eb5863b29de94 P 74d4625e810948d996703bbe0ff7c98d: stopping tablet replica
I20260929 20:59:28.477735 2038 raft_consensus.cc:2229] T 9bc05823db9f4760972eb5863b29de94 P 74d4625e810948d996703bbe0ff7c98d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:59:28.478308 2038 raft_consensus.cc:2258] T 9bc05823db9f4760972eb5863b29de94 P 74d4625e810948d996703bbe0ff7c98d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:59:28.479374 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 366816aa04ca40e2949b92c3f46d07cf (DEFAULT_TABLE table=test-table [id=f0e16c25daf44666a3ee58a06a74d2dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:59:28.480229 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 366816aa04ca40e2949b92c3f46d07cf (DEFAULT_TABLE table=test-table [id=f0e16c25daf44666a3ee58a06a74d2dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:59:28.480290 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 366816aa04ca40e2949b92c3f46d07cf (DEFAULT_TABLE table=test-table [id=f0e16c25daf44666a3ee58a06a74d2dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:59:28.480692 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 366816aa04ca40e2949b92c3f46d07cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:28.480801 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 366816aa04ca40e2949b92c3f46d07cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:28.481115 2036 ts_tablet_manager.cc:1905] T 9bc05823db9f4760972eb5863b29de94 P 5f1413b4f1864fa99efd5f319ff7d60b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:59:28.485594 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 366816aa04ca40e2949b92c3f46d07cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:28.487718 2016 tablet_bootstrap.cc:492] T 366816aa04ca40e2949b92c3f46d07cf P 47d0abd5486b4bfdbfb7059499ff1d34: Bootstrap starting.
I20260929 20:59:28.489431 2016 tablet_bootstrap.cc:654] T 366816aa04ca40e2949b92c3f46d07cf P 47d0abd5486b4bfdbfb7059499ff1d34: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:28.490597 2038 ts_tablet_manager.cc:1905] T 9bc05823db9f4760972eb5863b29de94 P 74d4625e810948d996703bbe0ff7c98d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:59:28.491516 2037 tablet_replica.cc:330] T 9bc05823db9f4760972eb5863b29de94 P 47d0abd5486b4bfdbfb7059499ff1d34: stopping tablet replica
I20260929 20:59:28.491919 2037 raft_consensus.cc:2229] T 9bc05823db9f4760972eb5863b29de94 P 47d0abd5486b4bfdbfb7059499ff1d34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:59:28.492427 2037 raft_consensus.cc:2258] T 9bc05823db9f4760972eb5863b29de94 P 47d0abd5486b4bfdbfb7059499ff1d34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:59:28.495694 2037 ts_tablet_manager.cc:1905] T 9bc05823db9f4760972eb5863b29de94 P 47d0abd5486b4bfdbfb7059499ff1d34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:59:28.496775 2014 tablet_bootstrap.cc:492] T 366816aa04ca40e2949b92c3f46d07cf P 74d4625e810948d996703bbe0ff7c98d: Bootstrap starting.
I20260929 20:59:28.498492 2016 tablet_bootstrap.cc:492] T 366816aa04ca40e2949b92c3f46d07cf P 47d0abd5486b4bfdbfb7059499ff1d34: No bootstrap required, opened a new log
I20260929 20:59:28.498792 2016 ts_tablet_manager.cc:1397] T 366816aa04ca40e2949b92c3f46d07cf P 47d0abd5486b4bfdbfb7059499ff1d34: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.001s
I20260929 20:59:28.499008 2014 tablet_bootstrap.cc:654] T 366816aa04ca40e2949b92c3f46d07cf P 74d4625e810948d996703bbe0ff7c98d: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:28.499882 2016 raft_consensus.cc:348] T 366816aa04ca40e2949b92c3f46d07cf P 47d0abd5486b4bfdbfb7059499ff1d34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } } peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } }
I20260929 20:59:28.500339 2016 raft_consensus.cc:374] T 366816aa04ca40e2949b92c3f46d07cf P 47d0abd5486b4bfdbfb7059499ff1d34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:28.500553 2016 raft_consensus.cc:729] T 366816aa04ca40e2949b92c3f46d07cf P 47d0abd5486b4bfdbfb7059499ff1d34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47d0abd5486b4bfdbfb7059499ff1d34, State: Initialized, Role: FOLLOWER
I20260929 20:59:28.501173 2016 consensus_queue.cc:260] T 366816aa04ca40e2949b92c3f46d07cf P 47d0abd5486b4bfdbfb7059499ff1d34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } } peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } }
I20260929 20:59:28.502382 2014 tablet_bootstrap.cc:492] T 366816aa04ca40e2949b92c3f46d07cf P 74d4625e810948d996703bbe0ff7c98d: No bootstrap required, opened a new log
I20260929 20:59:28.502594 2014 ts_tablet_manager.cc:1397] T 366816aa04ca40e2949b92c3f46d07cf P 74d4625e810948d996703bbe0ff7c98d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 20:59:28.504421 2016 ts_tablet_manager.cc:1428] T 366816aa04ca40e2949b92c3f46d07cf P 47d0abd5486b4bfdbfb7059499ff1d34: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260929 20:59:28.505053 2014 raft_consensus.cc:348] T 366816aa04ca40e2949b92c3f46d07cf P 74d4625e810948d996703bbe0ff7c98d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } } peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } }
I20260929 20:59:28.505355 2014 raft_consensus.cc:374] T 366816aa04ca40e2949b92c3f46d07cf P 74d4625e810948d996703bbe0ff7c98d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:28.506507 2014 raft_consensus.cc:729] T 366816aa04ca40e2949b92c3f46d07cf P 74d4625e810948d996703bbe0ff7c98d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74d4625e810948d996703bbe0ff7c98d, State: Initialized, Role: FOLLOWER
I20260929 20:59:28.506820 2014 consensus_queue.cc:260] T 366816aa04ca40e2949b92c3f46d07cf P 74d4625e810948d996703bbe0ff7c98d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } } peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } }
I20260929 20:59:28.507365 2014 ts_tablet_manager.cc:1428] T 366816aa04ca40e2949b92c3f46d07cf P 74d4625e810948d996703bbe0ff7c98d: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260929 20:59:28.566591 2038 ts_tablet_manager.cc:1918] T 9bc05823db9f4760972eb5863b29de94 P 74d4625e810948d996703bbe0ff7c98d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:59:28.569056 2038 log.cc:1196] T 9bc05823db9f4760972eb5863b29de94 P 74d4625e810948d996703bbe0ff7c98d: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-0/wal/wals/9bc05823db9f4760972eb5863b29de94
I20260929 20:59:28.570130 2038 ts_tablet_manager.cc:1939] T 9bc05823db9f4760972eb5863b29de94 P 74d4625e810948d996703bbe0ff7c98d: Deleting consensus metadata
I20260929 20:59:28.570804 2015 tablet_bootstrap.cc:492] T 366816aa04ca40e2949b92c3f46d07cf P 5f1413b4f1864fa99efd5f319ff7d60b: Bootstrap starting.
I20260929 20:59:28.572544 2015 tablet_bootstrap.cc:654] T 366816aa04ca40e2949b92c3f46d07cf P 5f1413b4f1864fa99efd5f319ff7d60b: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:28.574741 2015 tablet_bootstrap.cc:492] T 366816aa04ca40e2949b92c3f46d07cf P 5f1413b4f1864fa99efd5f319ff7d60b: No bootstrap required, opened a new log
I20260929 20:59:28.574888 2015 ts_tablet_manager.cc:1397] T 366816aa04ca40e2949b92c3f46d07cf P 5f1413b4f1864fa99efd5f319ff7d60b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260929 20:59:28.575716 2015 raft_consensus.cc:348] T 366816aa04ca40e2949b92c3f46d07cf P 5f1413b4f1864fa99efd5f319ff7d60b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } } peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } }
I20260929 20:59:28.575954 2015 raft_consensus.cc:374] T 366816aa04ca40e2949b92c3f46d07cf P 5f1413b4f1864fa99efd5f319ff7d60b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:28.576048 2015 raft_consensus.cc:729] T 366816aa04ca40e2949b92c3f46d07cf P 5f1413b4f1864fa99efd5f319ff7d60b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f1413b4f1864fa99efd5f319ff7d60b, State: Initialized, Role: FOLLOWER
I20260929 20:59:28.576328 2015 consensus_queue.cc:260] T 366816aa04ca40e2949b92c3f46d07cf P 5f1413b4f1864fa99efd5f319ff7d60b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } } peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } }
I20260929 20:59:28.576947 2015 ts_tablet_manager.cc:1428] T 366816aa04ca40e2949b92c3f46d07cf P 5f1413b4f1864fa99efd5f319ff7d60b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 20:59:28.634582 1495 catalog_manager.cc:4841] TS 74d4625e810948d996703bbe0ff7c98d (127.0.97.1:40541): tablet 9bc05823db9f4760972eb5863b29de94 (table test-table [id=1d7be766919f40b5af7776894a222a2e]) successfully deleted
I20260929 20:59:28.664692 2037 ts_tablet_manager.cc:1918] T 9bc05823db9f4760972eb5863b29de94 P 47d0abd5486b4bfdbfb7059499ff1d34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:59:28.664707 2036 ts_tablet_manager.cc:1918] T 9bc05823db9f4760972eb5863b29de94 P 5f1413b4f1864fa99efd5f319ff7d60b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:59:28.664934 2037 log.cc:1196] T 9bc05823db9f4760972eb5863b29de94 P 47d0abd5486b4bfdbfb7059499ff1d34: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-1/wal/wals/9bc05823db9f4760972eb5863b29de94
I20260929 20:59:28.665055 2036 log.cc:1196] T 9bc05823db9f4760972eb5863b29de94 P 5f1413b4f1864fa99efd5f319ff7d60b: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-2/wal/wals/9bc05823db9f4760972eb5863b29de94
I20260929 20:59:28.665508 2037 ts_tablet_manager.cc:1939] T 9bc05823db9f4760972eb5863b29de94 P 47d0abd5486b4bfdbfb7059499ff1d34: Deleting consensus metadata
I20260929 20:59:28.665853 2036 ts_tablet_manager.cc:1939] T 9bc05823db9f4760972eb5863b29de94 P 5f1413b4f1864fa99efd5f319ff7d60b: Deleting consensus metadata
I20260929 20:59:28.668226 1498 catalog_manager.cc:4841] TS 47d0abd5486b4bfdbfb7059499ff1d34 (127.0.97.2:35735): tablet 9bc05823db9f4760972eb5863b29de94 (table test-table [id=1d7be766919f40b5af7776894a222a2e]) successfully deleted
I20260929 20:59:28.668556 1497 catalog_manager.cc:4841] TS 5f1413b4f1864fa99efd5f319ff7d60b (127.0.97.3:41889): tablet 9bc05823db9f4760972eb5863b29de94 (table test-table [id=1d7be766919f40b5af7776894a222a2e]) successfully deleted
I20260929 20:59:28.716917 2021 raft_consensus.cc:482] T 366816aa04ca40e2949b92c3f46d07cf P 47d0abd5486b4bfdbfb7059499ff1d34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:59:28.717168 2021 raft_consensus.cc:504] T 366816aa04ca40e2949b92c3f46d07cf P 47d0abd5486b4bfdbfb7059499ff1d34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } } peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } }
I20260929 20:59:28.718629 2021 leader_election.cc:288] T 366816aa04ca40e2949b92c3f46d07cf P 47d0abd5486b4bfdbfb7059499ff1d34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74d4625e810948d996703bbe0ff7c98d (127.0.97.1:40541), 5f1413b4f1864fa99efd5f319ff7d60b (127.0.97.3:41889)
I20260929 20:59:28.726951 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:28.719034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51767 (local address 127.0.97.3:41889)
0929 20:59:28.719246 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:28.719256 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:28.720072 (+ 816us) server_negotiation.cc:407] Connection header received
0929 20:59:28.720328 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:28.720334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:28.720419 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:28.720518 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:59:28.721774 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:28.722879 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:28.724816 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:28.725910 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:28.726047 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:28.726149 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:28.726492 (+ 343us) server_negotiation.cc:299] Negotiation successful
0929 20:59:28.726640 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 20:59:28.727000 2042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:28.718988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41889 (local address 127.0.97.2:51767)
0929 20:59:28.719965 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:28.719984 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:28.720158 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:28.720696 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:28.720703 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:28.720733 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:59:28.721221 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:28.721232 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:28.723010 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:28.723017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:28.724640 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:28.724652 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:28.726087 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:28.726095 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:28.726307 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:28.726343 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 20:59:28.726451 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 20:59:28.726583 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":848,"mutex_wait_us":90,"thread_start_us":384,"threads_started":1}
I20260929 20:59:28.727711 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:28.718875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40541 (local address 127.0.97.2:53721)
0929 20:59:28.719640 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:28.719665 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:28.719861 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:28.720551 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:28.720556 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:28.720585 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:59:28.720989 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:28.720999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:28.722939 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:28.722946 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:28.725269 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:28.725284 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:28.726877 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:28.726883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:28.727090 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:28.727120 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 20:59:28.727205 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 20:59:28.727330 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":96,"thread_start_us":769,"threads_started":1}
I20260929 20:59:28.728307 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:28.719250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53721 (local address 127.0.97.1:40541)
0929 20:59:28.720289 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:28.720294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:28.720308 (+ 14us) server_negotiation.cc:407] Connection header received
0929 20:59:28.720373 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:28.720376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:28.720429 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:28.720537 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:59:28.721169 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:28.722776 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:28.725432 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:28.726515 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:28.726628 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:28.726730 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:28.727637 (+ 907us) server_negotiation.cc:299] Negotiation successful
0929 20:59:28.727969 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":897,"thread_start_us":789,"threads_started":1}
I20260929 20:59:28.729023 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "366816aa04ca40e2949b92c3f46d07cf" candidate_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74d4625e810948d996703bbe0ff7c98d" is_pre_election: true
I20260929 20:59:28.729288 1672 raft_consensus.cc:2454] T 366816aa04ca40e2949b92c3f46d07cf P 74d4625e810948d996703bbe0ff7c98d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47d0abd5486b4bfdbfb7059499ff1d34 in term 0.
I20260929 20:59:28.729140 1950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "366816aa04ca40e2949b92c3f46d07cf" candidate_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" is_pre_election: true
I20260929 20:59:28.729558 1950 raft_consensus.cc:2454] T 366816aa04ca40e2949b92c3f46d07cf P 5f1413b4f1864fa99efd5f319ff7d60b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47d0abd5486b4bfdbfb7059499ff1d34 in term 0.
I20260929 20:59:28.729867 1737 leader_election.cc:302] T 366816aa04ca40e2949b92c3f46d07cf P 47d0abd5486b4bfdbfb7059499ff1d34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47d0abd5486b4bfdbfb7059499ff1d34, 74d4625e810948d996703bbe0ff7c98d; no voters:
I20260929 20:59:28.730334 2021 raft_consensus.cc:2781] T 366816aa04ca40e2949b92c3f46d07cf P 47d0abd5486b4bfdbfb7059499ff1d34 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:59:28.730470 2021 raft_consensus.cc:482] T 366816aa04ca40e2949b92c3f46d07cf P 47d0abd5486b4bfdbfb7059499ff1d34 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:59:28.730603 2021 raft_consensus.cc:3037] T 366816aa04ca40e2949b92c3f46d07cf P 47d0abd5486b4bfdbfb7059499ff1d34 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:28.765393 2021 raft_consensus.cc:504] T 366816aa04ca40e2949b92c3f46d07cf P 47d0abd5486b4bfdbfb7059499ff1d34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } } peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } }
I20260929 20:59:28.766376 2021 leader_election.cc:288] T 366816aa04ca40e2949b92c3f46d07cf P 47d0abd5486b4bfdbfb7059499ff1d34 [CANDIDATE]: Term 1 election: Requested vote from peers 74d4625e810948d996703bbe0ff7c98d (127.0.97.1:40541), 5f1413b4f1864fa99efd5f319ff7d60b (127.0.97.3:41889)
I20260929 20:59:28.767007 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "366816aa04ca40e2949b92c3f46d07cf" candidate_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74d4625e810948d996703bbe0ff7c98d"
I20260929 20:59:28.767272 1672 raft_consensus.cc:3037] T 366816aa04ca40e2949b92c3f46d07cf P 74d4625e810948d996703bbe0ff7c98d [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:28.767252 1950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "366816aa04ca40e2949b92c3f46d07cf" candidate_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1413b4f1864fa99efd5f319ff7d60b"
I20260929 20:59:28.767505 1950 raft_consensus.cc:3037] T 366816aa04ca40e2949b92c3f46d07cf P 5f1413b4f1864fa99efd5f319ff7d60b [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:28.770967 1672 raft_consensus.cc:2454] T 366816aa04ca40e2949b92c3f46d07cf P 74d4625e810948d996703bbe0ff7c98d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47d0abd5486b4bfdbfb7059499ff1d34 in term 1.
I20260929 20:59:28.771473 1737 leader_election.cc:302] T 366816aa04ca40e2949b92c3f46d07cf P 47d0abd5486b4bfdbfb7059499ff1d34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47d0abd5486b4bfdbfb7059499ff1d34, 74d4625e810948d996703bbe0ff7c98d; no voters:
I20260929 20:59:28.771793 2021 raft_consensus.cc:2781] T 366816aa04ca40e2949b92c3f46d07cf P 47d0abd5486b4bfdbfb7059499ff1d34 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:59:28.772226 2021 raft_consensus.cc:686] T 366816aa04ca40e2949b92c3f46d07cf P 47d0abd5486b4bfdbfb7059499ff1d34 [term 1 LEADER]: Becoming Leader. State: Replica: 47d0abd5486b4bfdbfb7059499ff1d34, State: Running, Role: LEADER
I20260929 20:59:28.772948 2021 consensus_queue.cc:237] T 366816aa04ca40e2949b92c3f46d07cf P 47d0abd5486b4bfdbfb7059499ff1d34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } } peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } }
I20260929 20:59:28.776643 1511 catalog_manager.cc:5495] T 366816aa04ca40e2949b92c3f46d07cf P 47d0abd5486b4bfdbfb7059499ff1d34 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47d0abd5486b4bfdbfb7059499ff1d34 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } health_report { overall_health: UNKNOWN } } }
Sep 29 20:59:28 dist-test-slave-1dms krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:59:28.834242 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:28.815281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37173 (local address 127.0.0.1:54844)
0929 20:59:28.815965 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:28.815989 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:28.816146 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:28.816479 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:28.816483 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:28.817251 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:28.817427 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:28.817436 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:28.818499 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:28.818505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:28.819825 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:28.819835 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:28.820544 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:28.820549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:28.820745 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:28.825973 (+ 5228us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:59:28.825990 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:28.828522 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:28.831554 (+ 3032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:28.831566 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:28.831574 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:28.831999 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:28.832371 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:28.832377 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:28.832381 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:28.832511 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:28.833056 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:28.833063 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:28.833234 (+ 171us) client_negotiation.cc:769] Sending connection context
0929 20:59:28.833412 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 20:59:28.833741 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":22,"thread_start_us":358,"threads_started":1}
I20260929 20:59:28.834270 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:28.815319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54844 (local address 127.0.97.62:37173)
0929 20:59:28.816202 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:28.816208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:28.816243 (+ 35us) server_negotiation.cc:407] Connection header received
0929 20:59:28.816296 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:28.816299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:28.816347 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:28.816464 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:28.817576 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:28.818368 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:28.819958 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:28.820391 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:28.820529 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:28.828717 (+ 8188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:28.828723 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:28.828726 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:28.828753 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:28.831409 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:28.832128 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:28.832135 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:28.832141 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:28.832268 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:28.832607 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:28.832614 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:28.832617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:28.832944 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:28.833078 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:28.833482 (+ 404us) server_negotiation.cc:299] Negotiation successful
0929 20:59:28.833832 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":157,"server-negotiator.queue_time_us":767,"thread_start_us":638,"threads_started":1}
I20260929 20:59:28.840499 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54844:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 20:59:28.841614 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:59:28.849234 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 36c5d2a3f5f24e878532e84891b4e500 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a9f75c9fe97f43b4b4b40717caf1098a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:59:28.849519 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 36c5d2a3f5f24e878532e84891b4e500 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a9f75c9fe97f43b4b4b40717caf1098a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:59:28.849582 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 36c5d2a3f5f24e878532e84891b4e500 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a9f75c9fe97f43b4b4b40717caf1098a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:59:28.849850 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36c5d2a3f5f24e878532e84891b4e500. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:28.850108 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36c5d2a3f5f24e878532e84891b4e500. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:28.850107 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36c5d2a3f5f24e878532e84891b4e500. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:28.856575 2014 tablet_bootstrap.cc:492] T 36c5d2a3f5f24e878532e84891b4e500 P 74d4625e810948d996703bbe0ff7c98d: Bootstrap starting.
I20260929 20:59:28.857582 2016 tablet_bootstrap.cc:492] T 36c5d2a3f5f24e878532e84891b4e500 P 47d0abd5486b4bfdbfb7059499ff1d34: Bootstrap starting.
I20260929 20:59:28.858112 2014 tablet_bootstrap.cc:654] T 36c5d2a3f5f24e878532e84891b4e500 P 74d4625e810948d996703bbe0ff7c98d: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:28.859021 2016 tablet_bootstrap.cc:654] T 36c5d2a3f5f24e878532e84891b4e500 P 47d0abd5486b4bfdbfb7059499ff1d34: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:28.859898 2014 tablet_bootstrap.cc:492] T 36c5d2a3f5f24e878532e84891b4e500 P 74d4625e810948d996703bbe0ff7c98d: No bootstrap required, opened a new log
I20260929 20:59:28.860059 2014 ts_tablet_manager.cc:1397] T 36c5d2a3f5f24e878532e84891b4e500 P 74d4625e810948d996703bbe0ff7c98d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 20:59:28.860855 2014 raft_consensus.cc:348] T 36c5d2a3f5f24e878532e84891b4e500 P 74d4625e810948d996703bbe0ff7c98d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } } peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } }
I20260929 20:59:28.861070 2016 tablet_bootstrap.cc:492] T 36c5d2a3f5f24e878532e84891b4e500 P 47d0abd5486b4bfdbfb7059499ff1d34: No bootstrap required, opened a new log
I20260929 20:59:28.861079 2014 raft_consensus.cc:374] T 36c5d2a3f5f24e878532e84891b4e500 P 74d4625e810948d996703bbe0ff7c98d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:28.861150 2014 raft_consensus.cc:729] T 36c5d2a3f5f24e878532e84891b4e500 P 74d4625e810948d996703bbe0ff7c98d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74d4625e810948d996703bbe0ff7c98d, State: Initialized, Role: FOLLOWER
I20260929 20:59:28.861214 2016 ts_tablet_manager.cc:1397] T 36c5d2a3f5f24e878532e84891b4e500 P 47d0abd5486b4bfdbfb7059499ff1d34: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260929 20:59:28.861852 2016 raft_consensus.cc:348] T 36c5d2a3f5f24e878532e84891b4e500 P 47d0abd5486b4bfdbfb7059499ff1d34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } } peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } }
I20260929 20:59:28.861927 2014 consensus_queue.cc:260] T 36c5d2a3f5f24e878532e84891b4e500 P 74d4625e810948d996703bbe0ff7c98d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } } peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } }
I20260929 20:59:28.862068 2016 raft_consensus.cc:374] T 36c5d2a3f5f24e878532e84891b4e500 P 47d0abd5486b4bfdbfb7059499ff1d34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:28.862142 2016 raft_consensus.cc:729] T 36c5d2a3f5f24e878532e84891b4e500 P 47d0abd5486b4bfdbfb7059499ff1d34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47d0abd5486b4bfdbfb7059499ff1d34, State: Initialized, Role: FOLLOWER
I20260929 20:59:28.862398 2022 tablet_replica.cc:439] T 36c5d2a3f5f24e878532e84891b4e500 P 74d4625e810948d996703bbe0ff7c98d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } } peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } } }
I20260929 20:59:28.862547 2016 consensus_queue.cc:260] T 36c5d2a3f5f24e878532e84891b4e500 P 47d0abd5486b4bfdbfb7059499ff1d34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } } peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } }
I20260929 20:59:28.862789 2022 tablet_replica.cc:442] T 36c5d2a3f5f24e878532e84891b4e500 P 74d4625e810948d996703bbe0ff7c98d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:59:28.862929 2021 tablet_replica.cc:439] T 36c5d2a3f5f24e878532e84891b4e500 P 47d0abd5486b4bfdbfb7059499ff1d34: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } } peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } } }
I20260929 20:59:28.863181 2021 tablet_replica.cc:442] T 36c5d2a3f5f24e878532e84891b4e500 P 47d0abd5486b4bfdbfb7059499ff1d34: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:59:28.863200 2014 ts_tablet_manager.cc:1428] T 36c5d2a3f5f24e878532e84891b4e500 P 74d4625e810948d996703bbe0ff7c98d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 20:59:28.864696 2016 ts_tablet_manager.cc:1428] T 36c5d2a3f5f24e878532e84891b4e500 P 47d0abd5486b4bfdbfb7059499ff1d34: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 20:59:28.872063 1950 raft_consensus.cc:2454] T 366816aa04ca40e2949b92c3f46d07cf P 5f1413b4f1864fa99efd5f319ff7d60b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47d0abd5486b4bfdbfb7059499ff1d34 in term 1.
I20260929 20:59:28.875579 2015 tablet_bootstrap.cc:492] T 36c5d2a3f5f24e878532e84891b4e500 P 5f1413b4f1864fa99efd5f319ff7d60b: Bootstrap starting.
I20260929 20:59:28.877157 2015 tablet_bootstrap.cc:654] T 36c5d2a3f5f24e878532e84891b4e500 P 5f1413b4f1864fa99efd5f319ff7d60b: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:28.879957 2015 tablet_bootstrap.cc:492] T 36c5d2a3f5f24e878532e84891b4e500 P 5f1413b4f1864fa99efd5f319ff7d60b: No bootstrap required, opened a new log
I20260929 20:59:28.880298 2015 ts_tablet_manager.cc:1397] T 36c5d2a3f5f24e878532e84891b4e500 P 5f1413b4f1864fa99efd5f319ff7d60b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260929 20:59:28.881170 2015 raft_consensus.cc:348] T 36c5d2a3f5f24e878532e84891b4e500 P 5f1413b4f1864fa99efd5f319ff7d60b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } } peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } }
I20260929 20:59:28.881502 2015 raft_consensus.cc:374] T 36c5d2a3f5f24e878532e84891b4e500 P 5f1413b4f1864fa99efd5f319ff7d60b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:28.881637 2015 raft_consensus.cc:729] T 36c5d2a3f5f24e878532e84891b4e500 P 5f1413b4f1864fa99efd5f319ff7d60b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f1413b4f1864fa99efd5f319ff7d60b, State: Initialized, Role: FOLLOWER
I20260929 20:59:28.881918 2015 consensus_queue.cc:260] T 36c5d2a3f5f24e878532e84891b4e500 P 5f1413b4f1864fa99efd5f319ff7d60b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } } peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } }
I20260929 20:59:28.882395 2020 tablet_replica.cc:439] T 36c5d2a3f5f24e878532e84891b4e500 P 5f1413b4f1864fa99efd5f319ff7d60b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } } peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } } }
I20260929 20:59:28.882642 2020 tablet_replica.cc:442] T 36c5d2a3f5f24e878532e84891b4e500 P 5f1413b4f1864fa99efd5f319ff7d60b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:59:28.883602 2015 ts_tablet_manager.cc:1428] T 36c5d2a3f5f24e878532e84891b4e500 P 5f1413b4f1864fa99efd5f319ff7d60b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 20:59:29.037052 2021 raft_consensus.cc:482] T 36c5d2a3f5f24e878532e84891b4e500 P 47d0abd5486b4bfdbfb7059499ff1d34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:59:29.037276 2021 raft_consensus.cc:504] T 36c5d2a3f5f24e878532e84891b4e500 P 47d0abd5486b4bfdbfb7059499ff1d34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } } peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } }
I20260929 20:59:29.037851 2021 leader_election.cc:288] T 36c5d2a3f5f24e878532e84891b4e500 P 47d0abd5486b4bfdbfb7059499ff1d34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74d4625e810948d996703bbe0ff7c98d (127.0.97.1:40541), 5f1413b4f1864fa99efd5f319ff7d60b (127.0.97.3:41889)
I20260929 20:59:29.038455 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36c5d2a3f5f24e878532e84891b4e500" candidate_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74d4625e810948d996703bbe0ff7c98d" is_pre_election: true
I20260929 20:59:29.038455 1950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36c5d2a3f5f24e878532e84891b4e500" candidate_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" is_pre_election: true
I20260929 20:59:29.038689 1950 raft_consensus.cc:2454] T 36c5d2a3f5f24e878532e84891b4e500 P 5f1413b4f1864fa99efd5f319ff7d60b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47d0abd5486b4bfdbfb7059499ff1d34 in term 0.
I20260929 20:59:29.038710 1672 raft_consensus.cc:2454] T 36c5d2a3f5f24e878532e84891b4e500 P 74d4625e810948d996703bbe0ff7c98d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47d0abd5486b4bfdbfb7059499ff1d34 in term 0.
I20260929 20:59:29.039129 1737 leader_election.cc:302] T 36c5d2a3f5f24e878532e84891b4e500 P 47d0abd5486b4bfdbfb7059499ff1d34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 47d0abd5486b4bfdbfb7059499ff1d34, 5f1413b4f1864fa99efd5f319ff7d60b, 74d4625e810948d996703bbe0ff7c98d; no voters:
I20260929 20:59:29.039494 2021 raft_consensus.cc:2781] T 36c5d2a3f5f24e878532e84891b4e500 P 47d0abd5486b4bfdbfb7059499ff1d34 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:59:29.039568 2021 raft_consensus.cc:482] T 36c5d2a3f5f24e878532e84891b4e500 P 47d0abd5486b4bfdbfb7059499ff1d34 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:59:29.039691 2021 raft_consensus.cc:3037] T 36c5d2a3f5f24e878532e84891b4e500 P 47d0abd5486b4bfdbfb7059499ff1d34 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:29.067945 2021 raft_consensus.cc:504] T 36c5d2a3f5f24e878532e84891b4e500 P 47d0abd5486b4bfdbfb7059499ff1d34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } } peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } }
I20260929 20:59:29.068676 2021 leader_election.cc:288] T 36c5d2a3f5f24e878532e84891b4e500 P 47d0abd5486b4bfdbfb7059499ff1d34 [CANDIDATE]: Term 1 election: Requested vote from peers 74d4625e810948d996703bbe0ff7c98d (127.0.97.1:40541), 5f1413b4f1864fa99efd5f319ff7d60b (127.0.97.3:41889)
I20260929 20:59:29.069169 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36c5d2a3f5f24e878532e84891b4e500" candidate_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74d4625e810948d996703bbe0ff7c98d"
I20260929 20:59:29.069273 1950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36c5d2a3f5f24e878532e84891b4e500" candidate_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1413b4f1864fa99efd5f319ff7d60b"
I20260929 20:59:29.069425 1672 raft_consensus.cc:3037] T 36c5d2a3f5f24e878532e84891b4e500 P 74d4625e810948d996703bbe0ff7c98d [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:29.069494 1950 raft_consensus.cc:3037] T 36c5d2a3f5f24e878532e84891b4e500 P 5f1413b4f1864fa99efd5f319ff7d60b [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:29.169019 1672 raft_consensus.cc:2454] T 36c5d2a3f5f24e878532e84891b4e500 P 74d4625e810948d996703bbe0ff7c98d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47d0abd5486b4bfdbfb7059499ff1d34 in term 1.
I20260929 20:59:29.169023 1950 raft_consensus.cc:2454] T 36c5d2a3f5f24e878532e84891b4e500 P 5f1413b4f1864fa99efd5f319ff7d60b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47d0abd5486b4bfdbfb7059499ff1d34 in term 1.
I20260929 20:59:29.169569 1739 leader_election.cc:302] T 36c5d2a3f5f24e878532e84891b4e500 P 47d0abd5486b4bfdbfb7059499ff1d34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47d0abd5486b4bfdbfb7059499ff1d34, 5f1413b4f1864fa99efd5f319ff7d60b; no voters:
I20260929 20:59:29.169895 2021 raft_consensus.cc:2781] T 36c5d2a3f5f24e878532e84891b4e500 P 47d0abd5486b4bfdbfb7059499ff1d34 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:59:29.170363 2021 raft_consensus.cc:686] T 36c5d2a3f5f24e878532e84891b4e500 P 47d0abd5486b4bfdbfb7059499ff1d34 [term 1 LEADER]: Becoming Leader. State: Replica: 47d0abd5486b4bfdbfb7059499ff1d34, State: Running, Role: LEADER
I20260929 20:59:29.170691 2021 consensus_queue.cc:237] T 36c5d2a3f5f24e878532e84891b4e500 P 47d0abd5486b4bfdbfb7059499ff1d34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } } peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } }
I20260929 20:59:29.172072 2021 consensus_queue.cc:1035] T 366816aa04ca40e2949b92c3f46d07cf P 47d0abd5486b4bfdbfb7059499ff1d34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:59:29.173311 2057 tablet_replica.cc:439] T 36c5d2a3f5f24e878532e84891b4e500 P 47d0abd5486b4bfdbfb7059499ff1d34: TxnStatusTablet state changed. Reason: New leader 47d0abd5486b4bfdbfb7059499ff1d34. Latest consensus state: current_term: 1 leader_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } } peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } } }
I20260929 20:59:29.173647 2057 tablet_replica.cc:442] T 36c5d2a3f5f24e878532e84891b4e500 P 47d0abd5486b4bfdbfb7059499ff1d34: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:59:29.174070 1511 catalog_manager.cc:5495] T 36c5d2a3f5f24e878532e84891b4e500 P 47d0abd5486b4bfdbfb7059499ff1d34 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47d0abd5486b4bfdbfb7059499ff1d34 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } health_report { overall_health: UNKNOWN } } }
I20260929 20:59:29.177526 2059 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 20:59:29.177739 2059 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 20:59:29.181031 2021 consensus_queue.cc:1035] T 366816aa04ca40e2949b92c3f46d07cf P 47d0abd5486b4bfdbfb7059499ff1d34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:59:29.240396 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:29.233175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35735 (local address 127.0.0.1:58280)
0929 20:59:29.233379 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:29.233402 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:29.233560 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:29.234434 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:29.234440 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:29.234454 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:59:29.234655 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:29.234669 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:29.235743 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:29.235749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:29.237851 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:29.237866 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:29.238581 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:29.238595 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:29.238793 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:29.238865 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:59:29.239788 (+ 923us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:59:29.239808 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 20:59:29.239925 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 20:59:29.240099 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 20:59:29.240381 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:29.233319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58280 (local address 127.0.97.2:35735)
0929 20:59:29.234095 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:29.234100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:29.234113 (+ 13us) server_negotiation.cc:407] Connection header received
0929 20:59:29.234181 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:29.234185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:29.234256 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:29.234386 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:59:29.234808 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:29.235623 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:29.237981 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:29.238426 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:29.238523 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:29.239039 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:59:29.239595 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:59:29.239802 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:29.239983 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 20:59:29.240069 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":550,"threads_started":1}
I20260929 20:59:29.244105 1950 raft_consensus.cc:1261] T 36c5d2a3f5f24e878532e84891b4e500 P 5f1413b4f1864fa99efd5f319ff7d60b [term 1 FOLLOWER]: Refusing update from remote peer 47d0abd5486b4bfdbfb7059499ff1d34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:59:29.244143 1672 raft_consensus.cc:1261] T 36c5d2a3f5f24e878532e84891b4e500 P 74d4625e810948d996703bbe0ff7c98d [term 1 FOLLOWER]: Refusing update from remote peer 47d0abd5486b4bfdbfb7059499ff1d34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:59:29.244683 2021 consensus_queue.cc:1035] T 36c5d2a3f5f24e878532e84891b4e500 P 47d0abd5486b4bfdbfb7059499ff1d34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:59:29.245002 2045 consensus_queue.cc:1035] T 36c5d2a3f5f24e878532e84891b4e500 P 47d0abd5486b4bfdbfb7059499ff1d34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:59:29.247721 2022 tablet_replica.cc:439] T 36c5d2a3f5f24e878532e84891b4e500 P 74d4625e810948d996703bbe0ff7c98d: TxnStatusTablet state changed. Reason: New leader 47d0abd5486b4bfdbfb7059499ff1d34. Latest consensus state: current_term: 1 leader_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } } peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } } }
I20260929 20:59:29.247974 2022 tablet_replica.cc:442] T 36c5d2a3f5f24e878532e84891b4e500 P 74d4625e810948d996703bbe0ff7c98d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:59:29.249555 2061 mvcc.cc:205] Tried to move back new op lower bound from 7334770971618119680 to 7334770971325784064. Current Snapshot: MvccSnapshot[applied={T|T < 7334770971618119680}]
I20260929 20:59:29.253235 2057 tablet_replica.cc:439] T 36c5d2a3f5f24e878532e84891b4e500 P 47d0abd5486b4bfdbfb7059499ff1d34: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } } peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } } }
I20260929 20:59:29.253839 2057 tablet_replica.cc:442] T 36c5d2a3f5f24e878532e84891b4e500 P 47d0abd5486b4bfdbfb7059499ff1d34: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:59:29.259248 2022 tablet_replica.cc:439] T 36c5d2a3f5f24e878532e84891b4e500 P 74d4625e810948d996703bbe0ff7c98d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } } peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } } }
I20260929 20:59:29.259546 2022 tablet_replica.cc:442] T 36c5d2a3f5f24e878532e84891b4e500 P 74d4625e810948d996703bbe0ff7c98d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:59:29.263480 2020 tablet_replica.cc:439] T 36c5d2a3f5f24e878532e84891b4e500 P 5f1413b4f1864fa99efd5f319ff7d60b: TxnStatusTablet state changed. Reason: New leader 47d0abd5486b4bfdbfb7059499ff1d34. Latest consensus state: current_term: 1 leader_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } } peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } } }
I20260929 20:59:29.263788 2020 tablet_replica.cc:442] T 36c5d2a3f5f24e878532e84891b4e500 P 5f1413b4f1864fa99efd5f319ff7d60b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:59:29.263605 2045 tablet_replica.cc:439] T 36c5d2a3f5f24e878532e84891b4e500 P 47d0abd5486b4bfdbfb7059499ff1d34: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } } peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } } }
I20260929 20:59:29.263895 2045 tablet_replica.cc:442] T 36c5d2a3f5f24e878532e84891b4e500 P 47d0abd5486b4bfdbfb7059499ff1d34: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:59:29.276032 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:29.268649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58294 (local address 127.0.97.2:35735)
0929 20:59:29.269005 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:29.269012 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:29.269358 (+ 346us) server_negotiation.cc:407] Connection header received
0929 20:59:29.269563 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:29.269574 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:29.269664 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:29.269742 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:59:29.270342 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:29.271530 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:29.273626 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:29.274286 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:29.274388 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:29.274997 (+ 609us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:59:29.275123 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:59:29.275202 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:29.275649 (+ 447us) server_negotiation.cc:299] Negotiation successful
0929 20:59:29.275722 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220}
I20260929 20:59:29.277895 2020 tablet_replica.cc:439] T 36c5d2a3f5f24e878532e84891b4e500 P 5f1413b4f1864fa99efd5f319ff7d60b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d4625e810948d996703bbe0ff7c98d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40541 } } peers { permanent_uuid: "47d0abd5486b4bfdbfb7059499ff1d34" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35735 } } peers { permanent_uuid: "5f1413b4f1864fa99efd5f319ff7d60b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41889 } } }
I20260929 20:59:29.278136 2020 tablet_replica.cc:442] T 36c5d2a3f5f24e878532e84891b4e500 P 5f1413b4f1864fa99efd5f319ff7d60b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:59:29.292438 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:29.286501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58159 (local address 127.0.97.2:35735)
0929 20:59:29.286692 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:29.286700 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:29.287120 (+ 420us) server_negotiation.cc:407] Connection header received
0929 20:59:29.287319 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:29.287324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:29.287402 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:29.287511 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:59:29.288021 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:29.288792 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:29.290170 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:29.290963 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:29.291109 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:29.291590 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:59:29.291711 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:59:29.291840 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:29.291999 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 20:59:29.292062 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 20:59:29.292505 2077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:29.286435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35735 (local address 127.0.97.2:58159)
0929 20:59:29.287024 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:29.287043 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:29.287192 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:29.287515 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:29.287520 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:29.287530 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:59:29.287888 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:29.287896 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:29.288913 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:29.288919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:29.290022 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:29.290032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:29.291151 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:29.291158 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:29.291374 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:29.291428 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:59:29.291855 (+ 427us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:59:29.291867 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 20:59:29.291977 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 20:59:29.292105 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":238,"threads_started":1}
I20260929 20:59:29.361600 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:59:29.361846 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:59:29.365664 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3658792fe6c64331aead6d6f2097bc00: Sending DeleteTablet for 3 replicas of tablet 366816aa04ca40e2949b92c3f46d07cf
I20260929 20:59:29.366478 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet 366816aa04ca40e2949b92c3f46d07cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:59:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51220
I20260929 20:59:29.366755 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 366816aa04ca40e2949b92c3f46d07cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:59:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33254
I20260929 20:59:29.369994 1786 tablet_service.cc:1514] Processing DeleteTablet for tablet 366816aa04ca40e2949b92c3f46d07cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:59:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56834
I20260929 20:59:29.379957 2080 tablet_replica.cc:330] T 366816aa04ca40e2949b92c3f46d07cf P 5f1413b4f1864fa99efd5f319ff7d60b: stopping tablet replica
I20260929 20:59:29.380710 2080 raft_consensus.cc:2229] T 366816aa04ca40e2949b92c3f46d07cf P 5f1413b4f1864fa99efd5f319ff7d60b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:59:29.381090 2080 raft_consensus.cc:2258] T 366816aa04ca40e2949b92c3f46d07cf P 5f1413b4f1864fa99efd5f319ff7d60b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:59:29.382170 2080 ts_tablet_manager.cc:1905] T 366816aa04ca40e2949b92c3f46d07cf P 5f1413b4f1864fa99efd5f319ff7d60b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:59:29.385057 2081 tablet_replica.cc:330] T 366816aa04ca40e2949b92c3f46d07cf P 47d0abd5486b4bfdbfb7059499ff1d34: stopping tablet replica
I20260929 20:59:29.385365 2081 raft_consensus.cc:2229] T 366816aa04ca40e2949b92c3f46d07cf P 47d0abd5486b4bfdbfb7059499ff1d34 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:59:29.386075 2081 raft_consensus.cc:2258] T 366816aa04ca40e2949b92c3f46d07cf P 47d0abd5486b4bfdbfb7059499ff1d34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:59:29.387692 2081 ts_tablet_manager.cc:1905] T 366816aa04ca40e2949b92c3f46d07cf P 47d0abd5486b4bfdbfb7059499ff1d34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:59:29.388018 2079 tablet_replica.cc:330] T 366816aa04ca40e2949b92c3f46d07cf P 74d4625e810948d996703bbe0ff7c98d: stopping tablet replica
I20260929 20:59:29.388237 2079 raft_consensus.cc:2229] T 366816aa04ca40e2949b92c3f46d07cf P 74d4625e810948d996703bbe0ff7c98d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:59:29.388504 2079 raft_consensus.cc:2258] T 366816aa04ca40e2949b92c3f46d07cf P 74d4625e810948d996703bbe0ff7c98d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:59:29.391582 2079 ts_tablet_manager.cc:1905] T 366816aa04ca40e2949b92c3f46d07cf P 74d4625e810948d996703bbe0ff7c98d: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 20:59:29 dist-test-slave-1dms krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715567, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:59:29.418550 2083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:29.387818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39308 (local address 127.0.97.1:40541)
0929 20:59:29.392058 (+ 4240us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:29.392065 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:29.400005 (+ 7940us) server_negotiation.cc:407] Connection header received
0929 20:59:29.400179 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:29.400185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:29.400275 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:29.400396 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:29.402276 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:29.403377 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:29.405440 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:29.406089 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:29.406216 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:29.411126 (+ 4910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:29.411140 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:29.411159 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:29.411200 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:29.415344 (+ 4144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:29.416203 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:29.416216 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:29.416222 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:29.416369 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:29.416760 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:29.416763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:29.416765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:29.417173 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:29.417824 (+ 651us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:29.417951 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 20:59:29.418161 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4127,"thread_start_us":4034,"threads_started":1}
W20260929 20:59:29.419293 1712 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39308
I20260929 20:59:29.437101 2087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:29.422277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54850 (local address 127.0.97.62:37173)
0929 20:59:29.423210 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:29.423214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:29.423229 (+ 15us) server_negotiation.cc:407] Connection header received
0929 20:59:29.423273 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:29.423277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:29.423330 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:29.423413 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:29.425101 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:29.425924 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:29.427651 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:29.428076 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:29.428171 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:29.431353 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:29.431358 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:29.431361 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:29.431396 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:29.434294 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:29.435035 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:29.435039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:29.435041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:29.435153 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:29.435587 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:29.435590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:29.435592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:29.435839 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:29.436017 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:29.436390 (+ 373us) server_negotiation.cc:299] Negotiation successful
0929 20:59:29.436694 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":824,"thread_start_us":681,"threads_started":1}
W20260929 20:59:29.437839 1511 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54850
I20260929 20:59:29.440433 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 1582
I20260929 20:59:29.457556 2080 ts_tablet_manager.cc:1918] T 366816aa04ca40e2949b92c3f46d07cf P 5f1413b4f1864fa99efd5f319ff7d60b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:59:29.457808 2080 log.cc:1196] T 366816aa04ca40e2949b92c3f46d07cf P 5f1413b4f1864fa99efd5f319ff7d60b: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-2/wal/wals/366816aa04ca40e2949b92c3f46d07cf
I20260929 20:59:29.458278 2080 ts_tablet_manager.cc:1939] T 366816aa04ca40e2949b92c3f46d07cf P 5f1413b4f1864fa99efd5f319ff7d60b: Deleting consensus metadata
I20260929 20:59:29.458359 2081 ts_tablet_manager.cc:1918] T 366816aa04ca40e2949b92c3f46d07cf P 47d0abd5486b4bfdbfb7059499ff1d34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:59:29.460736 2083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:29.441287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39318 (local address 127.0.97.1:40541)
0929 20:59:29.441492 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:29.441498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:29.442420 (+ 922us) server_negotiation.cc:407] Connection header received
0929 20:59:29.442512 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:29.442516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:29.443013 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:29.443129 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:29.444272 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:29.445360 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:29.446754 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:29.447392 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:29.447520 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:29.449900 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:29.449906 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:29.449910 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:29.449948 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:29.453252 (+ 3304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:29.453831 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:29.453836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:29.453839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:29.458646 (+ 4807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:29.459070 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:29.459086 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:29.459093 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:29.459383 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:29.459599 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:29.460111 (+ 512us) server_negotiation.cc:299] Negotiation successful
0929 20:59:29.460362 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:59:29.462347 2081 log.cc:1196] T 366816aa04ca40e2949b92c3f46d07cf P 47d0abd5486b4bfdbfb7059499ff1d34: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790715556572573-388-0/minicluster-data/ts-1/wal/wals/366816aa04ca40e2949b92c3f46d07cf
W20260929 20:59:29.462666 1712 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39318
I20260929 20:59:29.462821 2081 ts_tablet_manager.cc:1939] T 366816aa04ca40e2949b92c3f46d07cf P 47d0abd5486b4bfdbfb7059499ff1d34: Deleting consensus metadata
W20260929 20:59:29.463210 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 1582: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:59:29.463414 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 1582
I20260929 20:59:29.510805 1497 catalog_manager.cc:4841] TS 5f1413b4f1864fa99efd5f319ff7d60b (127.0.97.3:41889): tablet 366816aa04ca40e2949b92c3f46d07cf (table test-table [id=f0e16c25daf44666a3ee58a06a74d2dd]) successfully deleted
I20260929 20:59:29.539083 1498 catalog_manager.cc:4841] TS 47d0abd5486b4bfdbfb7059499ff1d34 (127.0.97.2:35735): tablet 366816aa04ca40e2949b92c3f46d07cf (table test-table [id=f0e16c25daf44666a3ee58a06a74d2dd]) successfully deleted
W20260929 20:59:29.571565 1495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:40541 (error 108)
I20260929 20:59:29.571894 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 1721
I20260929 20:59:29.574704 2090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:29.573825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40541 (local address 127.0.0.1:39324)
0929 20:59:29.574357 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:29.574476 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":305,"threads_started":1}
W20260929 20:59:29.574967 1495 catalog_manager.cc:4584] TS 74d4625e810948d996703bbe0ff7c98d (127.0.97.1:40541): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 366816aa04ca40e2949b92c3f46d07cf: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40541: connect: Connection refused (error 111)
Sep 29 20:59:29 dist-test-slave-1dms krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715567, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:59:29.590382 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:29.573103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58300 (local address 127.0.97.2:35735)
0929 20:59:29.573317 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:29.573322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:29.573341 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:59:29.573393 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:29.573397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:29.573464 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:29.573552 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:29.575056 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:29.575878 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:29.577730 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:29.578167 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:29.578270 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:29.583038 (+ 4768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:29.583054 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:29.583073 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:29.583138 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:29.587242 (+ 4104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:29.588015 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:29.588025 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:29.588028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:29.588247 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:29.588660 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:29.588666 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:29.588669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:29.589190 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:29.589556 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:29.589725 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 20:59:29.589961 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 20:59:29.591212 1851 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58300
W20260929 20:59:29.591934 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 1721: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:59:29.592119 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 1721
I20260929 20:59:29.614701 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 1860
Sep 29 20:59:29 dist-test-slave-1dms krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715567, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:59:29.628402 2090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:29.627975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40541 (local address 127.0.0.1:39326)
0929 20:59:29.628175 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:29.628260 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 20:59:29.629592 2090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:29.629229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40541 (local address 127.0.0.1:39342)
0929 20:59:29.629396 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:29.629468 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 20:59:29.633394 2091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:29.615544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43348 (local address 127.0.97.3:41889)
0929 20:59:29.616281 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:29.616290 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:29.616316 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:59:29.616397 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:29.616408 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:29.616510 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:29.616669 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:29.617880 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:29.619228 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:29.620409 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:29.621248 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:29.621392 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:29.625509 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:29.625525 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:29.625558 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:29.625648 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:29.630311 (+ 4663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:29.631042 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:29.631056 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:29.631062 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:29.631270 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:29.631625 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:29.631632 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:29.631637 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:29.632148 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:29.632462 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:29.632630 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 20:59:29.632899 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":377,"threads_started":1}
W20260929 20:59:29.634194 1990 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43348
W20260929 20:59:29.634868 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 1860: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:59:29.635077 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 1860
I20260929 20:59:29.656987 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 1479
I20260929 20:59:29.658231 1551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:59:29.771266 2090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:29.770791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40541 (local address 127.0.0.1:39356)
0929 20:59:29.770989 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:29.771092 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 20:59:29.771376 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 1479
2026-09-29T20:59:29Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5128 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:59:29 dist-test-slave-1dms krb5kdc[2095](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:59:29 dist-test-slave-1dms krb5kdc[2095](info): set up 2 sockets
Sep 29 20:59:29 dist-test-slave-1dms krb5kdc[2095](info): commencing operation
krb5kdc: starting...
W20260929 20:59:30.848955 388 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.007s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 20:59:30 dist-test-slave-1dms krb5kdc[2095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:59:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:59:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:59:31.030822 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40803
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:59:31.341760 2108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:31.342036 2108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:31.342217 2108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:31.349399 2108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:59:31.349493 2108 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 20:59:31.349601 2108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:31.349704 2108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:31.349790 2108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:59:31.349905 2108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:59:31.359392 2108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40803
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:31.360707 2108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:31.362027 2108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:31.373953 2116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:31.373908 2114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:31.373975 2113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:31.375069 2108 server_base.cc:1034] running on GCE node
I20260929 20:59:31.375842 2108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:31.377306 2108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:31.378538 2108 hybrid_clock.cc:648] HybridClock initialized: now 1790715571378465 us; error 41 us; skew 500 ppm
I20260929 20:59:31.378886 2108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:59:31 dist-test-slave-1dms krb5kdc[2095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:59:31.381975 2108 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:59:31.383422 2108 webserver.cc:460] Webserver started at http://127.0.97.62:37225/ using document root <none> and password file <none>
I20260929 20:59:31.384100 2108 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:31.384224 2108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:31.384534 2108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:31.387399 2108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "d36341f2e2ba4bccbbae2e9c386e6696"
format_stamp: "Formatted at 2026-09-29 20:59:31 on dist-test-slave-1dms"
server_key: "22ce47b0cfa714648ade5bfe5fc6c7b9"
server_key_iv: "8de8bebb111c4a51101da714fb8d2251"
server_key_version: "encryptionkey@0"
I20260929 20:59:31.388130 2108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "d36341f2e2ba4bccbbae2e9c386e6696"
format_stamp: "Formatted at 2026-09-29 20:59:31 on dist-test-slave-1dms"
server_key: "22ce47b0cfa714648ade5bfe5fc6c7b9"
server_key_iv: "8de8bebb111c4a51101da714fb8d2251"
server_key_version: "encryptionkey@0"
I20260929 20:59:31.393294 2108 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 20:59:31.396472 2123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:31.397771 2108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:59:31.397945 2108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "d36341f2e2ba4bccbbae2e9c386e6696"
format_stamp: "Formatted at 2026-09-29 20:59:31 on dist-test-slave-1dms"
server_key: "22ce47b0cfa714648ade5bfe5fc6c7b9"
server_key_iv: "8de8bebb111c4a51101da714fb8d2251"
server_key_version: "encryptionkey@0"
I20260929 20:59:31.398154 2108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:31.415293 2108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:31.419224 2108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:31.419492 2108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:31.441087 2108 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40803
I20260929 20:59:31.441094 2184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40803 every 8 connection(s)
I20260929 20:59:31.442404 2108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:59:31.446236 2185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:31.452780 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 2108
I20260929 20:59:31.453049 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 20:59:31.453824 388 external_mini_cluster.cc:1428] Setting key 08e46d9ae58d3e4ea0f471d475eced93
I20260929 20:59:31.457396 2185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d36341f2e2ba4bccbbae2e9c386e6696: Bootstrap starting.
I20260929 20:59:31.460891 2185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d36341f2e2ba4bccbbae2e9c386e6696: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:31.461982 2185 log.cc:824] T 00000000000000000000000000000000 P d36341f2e2ba4bccbbae2e9c386e6696: Log is configured to *not* fsync() on all Append() calls
I20260929 20:59:31.464986 2185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d36341f2e2ba4bccbbae2e9c386e6696: No bootstrap required, opened a new log
Sep 29 20:59:31 dist-test-slave-1dms krb5kdc[2095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:59:31.473218 2185 raft_consensus.cc:348] T 00000000000000000000000000000000 P d36341f2e2ba4bccbbae2e9c386e6696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36341f2e2ba4bccbbae2e9c386e6696" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40803 } }
I20260929 20:59:31.473580 2185 raft_consensus.cc:374] T 00000000000000000000000000000000 P d36341f2e2ba4bccbbae2e9c386e6696 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:31.473681 2185 raft_consensus.cc:729] T 00000000000000000000000000000000 P d36341f2e2ba4bccbbae2e9c386e6696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d36341f2e2ba4bccbbae2e9c386e6696, State: Initialized, Role: FOLLOWER
I20260929 20:59:31.474371 2185 consensus_queue.cc:260] T 00000000000000000000000000000000 P d36341f2e2ba4bccbbae2e9c386e6696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36341f2e2ba4bccbbae2e9c386e6696" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40803 } }
I20260929 20:59:31.474655 2185 raft_consensus.cc:388] T 00000000000000000000000000000000 P d36341f2e2ba4bccbbae2e9c386e6696 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:59:31.474804 2185 raft_consensus.cc:482] T 00000000000000000000000000000000 P d36341f2e2ba4bccbbae2e9c386e6696 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:59:31.475018 2185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d36341f2e2ba4bccbbae2e9c386e6696 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:31.476969 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:31.455887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52706 (local address 127.0.97.62:40803)
0929 20:59:31.456670 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:31.456683 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:31.456723 (+ 40us) server_negotiation.cc:407] Connection header received
0929 20:59:31.458529 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:31.458551 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:31.458973 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:31.459313 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:31.461128 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:31.462515 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:31.464113 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:31.464744 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:31.464930 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:31.469441 (+ 4511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:31.469451 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:31.469464 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:31.469504 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:31.472482 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:31.473197 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:31.473207 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:31.473213 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:31.473363 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:31.473787 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:31.473793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:31.473798 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:31.474454 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:31.474749 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:31.475181 (+ 432us) server_negotiation.cc:299] Negotiation successful
0929 20:59:31.475607 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":378,"threads_started":1}
I20260929 20:59:31.478260 2185 raft_consensus.cc:504] T 00000000000000000000000000000000 P d36341f2e2ba4bccbbae2e9c386e6696 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36341f2e2ba4bccbbae2e9c386e6696" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40803 } }
I20260929 20:59:31.478761 2185 leader_election.cc:302] T 00000000000000000000000000000000 P d36341f2e2ba4bccbbae2e9c386e6696 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d36341f2e2ba4bccbbae2e9c386e6696; no voters:
I20260929 20:59:31.479336 2185 leader_election.cc:288] T 00000000000000000000000000000000 P d36341f2e2ba4bccbbae2e9c386e6696 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:59:31.479480 2190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d36341f2e2ba4bccbbae2e9c386e6696 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:59:31.480448 2190 raft_consensus.cc:686] T 00000000000000000000000000000000 P d36341f2e2ba4bccbbae2e9c386e6696 [term 1 LEADER]: Becoming Leader. State: Replica: d36341f2e2ba4bccbbae2e9c386e6696, State: Running, Role: LEADER
I20260929 20:59:31.480953 2190 consensus_queue.cc:237] T 00000000000000000000000000000000 P d36341f2e2ba4bccbbae2e9c386e6696 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36341f2e2ba4bccbbae2e9c386e6696" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40803 } }
I20260929 20:59:31.481716 2185 sys_catalog.cc:564] T 00000000000000000000000000000000 P d36341f2e2ba4bccbbae2e9c386e6696 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:59:31.483789 2191 sys_catalog.cc:455] T 00000000000000000000000000000000 P d36341f2e2ba4bccbbae2e9c386e6696 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d36341f2e2ba4bccbbae2e9c386e6696" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36341f2e2ba4bccbbae2e9c386e6696" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40803 } } }
I20260929 20:59:31.484131 2191 sys_catalog.cc:458] T 00000000000000000000000000000000 P d36341f2e2ba4bccbbae2e9c386e6696 [sys.catalog]: This master's current role is: LEADER
I20260929 20:59:31.484185 2192 sys_catalog.cc:455] T 00000000000000000000000000000000 P d36341f2e2ba4bccbbae2e9c386e6696 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d36341f2e2ba4bccbbae2e9c386e6696. Latest consensus state: current_term: 1 leader_uuid: "d36341f2e2ba4bccbbae2e9c386e6696" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36341f2e2ba4bccbbae2e9c386e6696" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40803 } } }
I20260929 20:59:31.484444 2192 sys_catalog.cc:458] T 00000000000000000000000000000000 P d36341f2e2ba4bccbbae2e9c386e6696 [sys.catalog]: This master's current role is: LEADER
I20260929 20:59:31.485251 2197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:59:31.491413 2197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:59:31.500694 2207 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 20:59:31.503228 2197 catalog_manager.cc:1347] Generated new cluster ID: 190183e0cb8045ebba7ea574b67f49a1
I20260929 20:59:31.503329 2197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:59:31.510316 2197 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:59:31.511986 2197 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:59:31.522934 2197 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d36341f2e2ba4bccbbae2e9c386e6696: Generated new TSK 0
I20260929 20:59:31.523789 2197 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:59:31.581230 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40803
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:59:31.910197 2211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:31.910470 2211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:31.910569 2211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:31.917709 2211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:31.917814 2211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:31.917896 2211 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 20:59:31.917948 2211 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:59:31.918016 2211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:59:31.928273 2211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:31.929663 2211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:31.930940 2211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:31.943657 2219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:31.943615 2216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:31.943679 2217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:31.944442 2211 server_base.cc:1034] running on GCE node
I20260929 20:59:31.945087 2211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:31.946071 2211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:31.947256 2211 hybrid_clock.cc:648] HybridClock initialized: now 1790715571947216 us; error 32 us; skew 500 ppm
I20260929 20:59:31.947553 2211 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:59:31 dist-test-slave-1dms krb5kdc[2095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:59:31.950652 2211 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:59:31.951933 2211 webserver.cc:460] Webserver started at http://127.0.97.1:43615/ using document root <none> and password file <none>
I20260929 20:59:31.952530 2211 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:31.952622 2211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:31.952943 2211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:31.955628 2211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-0/data/instance:
uuid: "b44f8fae008f4acf83a4a8b7aef1a2de"
format_stamp: "Formatted at 2026-09-29 20:59:31 on dist-test-slave-1dms"
server_key: "faa2da1b3a4c90d381b3409a2f0bf36b"
server_key_iv: "ee47bdb7d000187af514a7e42db1fd1f"
server_key_version: "encryptionkey@0"
I20260929 20:59:31.956314 2211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b44f8fae008f4acf83a4a8b7aef1a2de"
format_stamp: "Formatted at 2026-09-29 20:59:31 on dist-test-slave-1dms"
server_key: "faa2da1b3a4c90d381b3409a2f0bf36b"
server_key_iv: "ee47bdb7d000187af514a7e42db1fd1f"
server_key_version: "encryptionkey@0"
I20260929 20:59:31.961089 2211 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.003s
I20260929 20:59:31.964258 2226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:31.965610 2211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:59:31.965785 2211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-0/wal
uuid: "b44f8fae008f4acf83a4a8b7aef1a2de"
format_stamp: "Formatted at 2026-09-29 20:59:31 on dist-test-slave-1dms"
server_key: "faa2da1b3a4c90d381b3409a2f0bf36b"
server_key_iv: "ee47bdb7d000187af514a7e42db1fd1f"
server_key_version: "encryptionkey@0"
I20260929 20:59:31.965976 2211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:31.986764 2211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:31.990870 2211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:31.991150 2211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:31.993667 2211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:59:31.993803 2211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:31.993938 2211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:59:31.994067 2211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 29 20:59:32 dist-test-slave-1dms krb5kdc[2095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:59:32.016944 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:31.994108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34989 (local address 127.0.97.62:40803)
0929 20:59:31.994565 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:31.994572 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:31.995841 (+ 1269us) server_negotiation.cc:407] Connection header received
0929 20:59:31.998065 (+ 2224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:31.998070 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:31.998152 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:31.998286 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:32.000461 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:32.001776 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:32.004461 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:32.005224 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:32.005391 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:32.009464 (+ 4073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:32.009470 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:32.009476 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:32.009519 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:32.012750 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:32.013696 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:32.013702 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:32.013706 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:32.013846 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:32.014434 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:32.014442 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:32.014445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:32.014844 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:32.014980 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:32.016076 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0929 20:59:32.016385 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":240,"threads_started":1}
I20260929 20:59:32.017654 2236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:31.994781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40803 (local address 127.0.97.1:34989)
0929 20:59:31.995664 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:31.995710 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:31.997804 (+ 2094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:31.998538 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:31.998552 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:31.999343 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:32.000192 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:32.000213 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:32.001956 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:32.001963 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:32.004266 (+ 2303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:32.004304 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:32.005527 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:32.005534 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:32.005726 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:32.006459 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:59:32.006483 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:32.009147 (+ 2664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:32.012905 (+ 3758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:32.012917 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:32.012934 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:32.013523 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:32.013966 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:32.013970 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:32.013993 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:32.014308 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:32.015050 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:32.015056 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:32.015592 (+ 536us) client_negotiation.cc:769] Sending connection context
0929 20:59:32.015980 (+ 388us) client_negotiation.cc:240] Negotiation successful
0929 20:59:32.016394 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":394,"threads_started":1}
I20260929 20:59:32.036659 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:32.023340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50181 (local address 127.0.97.62:40803)
0929 20:59:32.023544 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:32.023552 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:32.024079 (+ 527us) server_negotiation.cc:407] Connection header received
0929 20:59:32.024214 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:32.024222 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:32.024325 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:32.024443 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:32.025335 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:32.026770 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:32.028447 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:32.029209 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:32.029364 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:32.031131 (+ 1767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:32.031135 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:32.031139 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:32.031167 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:32.033867 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:32.034723 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:32.034729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:32.034732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:32.034861 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:32.035309 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:32.035316 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:32.035321 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:32.035613 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:32.035783 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:32.036289 (+ 506us) server_negotiation.cc:299] Negotiation successful
0929 20:59:32.036440 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 20:59:32.037279 2292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:32.023336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40803 (local address 127.0.97.1:50181)
0929 20:59:32.023977 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:32.023995 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:32.024106 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:32.024470 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:32.024473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:32.024918 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:32.025190 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:32.025199 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:32.026965 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:32.026969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:32.028198 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:32.028208 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:32.029458 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:32.029464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:32.029589 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:32.030201 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:59:32.030217 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:32.030924 (+ 707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:32.034030 (+ 3106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:32.034041 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:32.034048 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:32.034549 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:32.034975 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:32.034981 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:32.034986 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:32.035159 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:32.035753 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:32.035759 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:32.036010 (+ 251us) client_negotiation.cc:769] Sending connection context
0929 20:59:32.036212 (+ 202us) client_negotiation.cc:240] Negotiation successful
0929 20:59:32.036635 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":369,"threads_started":1}
I20260929 20:59:32.051614 2211 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45601
I20260929 20:59:32.051656 2346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45601 every 8 connection(s)
I20260929 20:59:32.052800 2211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:59:32.054752 2347 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40803
I20260929 20:59:32.054973 2347 heartbeater.cc:464] Registering TS with master...
I20260929 20:59:32.055590 2347 heartbeater.cc:511] Master 127.0.97.62:40803 requested a full tablet report, sending...
I20260929 20:59:32.057502 2140 ts_manager.cc:194] Registered new tserver with Master: b44f8fae008f4acf83a4a8b7aef1a2de (127.0.97.1:45601)
I20260929 20:59:32.059437 2140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34989
I20260929 20:59:32.061059 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 2211
I20260929 20:59:32.061275 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:59:32.061853 388 external_mini_cluster.cc:1428] Setting key d088f0311066baf9ab996ab00521d941
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:59:32.100495 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40803
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:59:32.415256 2351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:32.415530 2351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:32.415750 2351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:32.423345 2351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:32.423489 2351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:32.423647 2351 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 20:59:32.423739 2351 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:59:32.423852 2351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:59:32.433830 2351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:32.435160 2351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:32.436484 2351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:32.450433 2356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:32.450881 2357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:32.451298 2351 server_base.cc:1034] running on GCE node
W20260929 20:59:32.453379 2359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:32.454123 2351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:32.455243 2351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:32.456478 2351 hybrid_clock.cc:648] HybridClock initialized: now 1790715572456407 us; error 65 us; skew 500 ppm
I20260929 20:59:32.456878 2351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:59:32 dist-test-slave-1dms krb5kdc[2095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:59:32.460405 2351 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:59:32.461859 2351 webserver.cc:460] Webserver started at http://127.0.97.2:34201/ using document root <none> and password file <none>
I20260929 20:59:32.462540 2351 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:32.462669 2351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:32.462973 2351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:32.465857 2351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-1/data/instance:
uuid: "878ec25593c146c9bdc3700d0fc30778"
format_stamp: "Formatted at 2026-09-29 20:59:32 on dist-test-slave-1dms"
server_key: "9c296cfdf8cc462b1d94d8f4b8c49ac8"
server_key_iv: "b7cfe5059df560740b54a497fff87b35"
server_key_version: "encryptionkey@0"
I20260929 20:59:32.466622 2351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-1/wal/instance:
uuid: "878ec25593c146c9bdc3700d0fc30778"
format_stamp: "Formatted at 2026-09-29 20:59:32 on dist-test-slave-1dms"
server_key: "9c296cfdf8cc462b1d94d8f4b8c49ac8"
server_key_iv: "b7cfe5059df560740b54a497fff87b35"
server_key_version: "encryptionkey@0"
I20260929 20:59:32.471737 2351 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:59:32.475116 2366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:32.476400 2351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 20:59:32.476572 2351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-1/wal
uuid: "878ec25593c146c9bdc3700d0fc30778"
format_stamp: "Formatted at 2026-09-29 20:59:32 on dist-test-slave-1dms"
server_key: "9c296cfdf8cc462b1d94d8f4b8c49ac8"
server_key_iv: "b7cfe5059df560740b54a497fff87b35"
server_key_version: "encryptionkey@0"
I20260929 20:59:32.476733 2351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:32.515133 2351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:32.520409 2351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:32.520704 2351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:32.523386 2351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:59:32.523478 2351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:32.523624 2351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:59:32.523705 2351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:59:32 dist-test-slave-1dms krb5kdc[2095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:59:32.546667 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:32.524077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33921 (local address 127.0.97.62:40803)
0929 20:59:32.524248 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:32.524254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:32.525798 (+ 1544us) server_negotiation.cc:407] Connection header received
0929 20:59:32.528950 (+ 3152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:32.528954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:32.529019 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:32.529133 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:32.531416 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:32.532429 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:32.535030 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:32.535602 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:32.535732 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:32.539851 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:32.539857 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:32.539863 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:32.539900 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:32.543234 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:32.544188 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:32.544193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:32.544197 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:32.544354 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:32.544950 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:32.544955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:32.544959 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:32.545244 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:32.545396 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:32.546110 (+ 714us) server_negotiation.cc:299] Negotiation successful
0929 20:59:32.546327 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:59:32.547755 2375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:32.524674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40803 (local address 127.0.97.2:33921)
0929 20:59:32.525609 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:32.525665 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:32.528539 (+ 2874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:32.529440 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:32.529454 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:32.530323 (+ 869us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:32.531234 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:32.531264 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:32.532603 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:32.532607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:32.534888 (+ 2281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:32.534900 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:32.536037 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:32.536048 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:32.536249 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:32.536966 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:59:32.536989 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:32.539645 (+ 2656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:32.543356 (+ 3711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:32.543367 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:32.543383 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:32.544027 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:32.544512 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:32.544518 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:32.544524 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:32.544778 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:32.545369 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:32.545376 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:32.545718 (+ 342us) client_negotiation.cc:769] Sending connection context
0929 20:59:32.545981 (+ 263us) client_negotiation.cc:240] Negotiation successful
0929 20:59:32.546492 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":415,"threads_started":1}
I20260929 20:59:32.568377 2432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:32.554188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40803 (local address 127.0.97.2:42671)
0929 20:59:32.555096 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:32.555121 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:32.555280 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:32.555595 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:32.555603 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:32.556351 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:32.556787 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:32.556803 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:32.558077 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:32.558083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:32.559061 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:32.559071 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:32.559800 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:32.559808 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:32.559974 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:32.560593 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:59:32.560610 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:32.561606 (+ 996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:32.565653 (+ 4047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:32.565661 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:32.565667 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:32.566260 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:32.566684 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:32.566687 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:32.566689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:32.566803 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:32.567572 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:32.567574 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:32.567726 (+ 152us) client_negotiation.cc:769] Sending connection context
0929 20:59:32.567834 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 20:59:32.568059 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":535,"threads_started":1}
I20260929 20:59:32.568771 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:32.554611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42671 (local address 127.0.97.62:40803)
0929 20:59:32.554805 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:32.554812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:32.555217 (+ 405us) server_negotiation.cc:407] Connection header received
0929 20:59:32.555398 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:32.555401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:32.555458 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:32.555535 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:32.556994 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:32.557950 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:32.559195 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:32.559612 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:32.559743 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:32.561789 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:32.561795 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:32.561800 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:32.561839 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:32.565489 (+ 3650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:32.566396 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:32.566401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:32.566404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:32.566563 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:32.566912 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:32.566920 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:32.566925 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:32.567451 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:32.567635 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:32.567990 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 20:59:32.568281 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 20:59:32.582227 2351 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33557
I20260929 20:59:32.582268 2485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33557 every 8 connection(s)
I20260929 20:59:32.583537 2351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:59:32.585574 2486 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40803
I20260929 20:59:32.585753 2486 heartbeater.cc:464] Registering TS with master...
I20260929 20:59:32.586354 2486 heartbeater.cc:511] Master 127.0.97.62:40803 requested a full tablet report, sending...
I20260929 20:59:32.587181 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 2351
I20260929 20:59:32.587366 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:59:32.587688 2140 ts_manager.cc:194] Registered new tserver with Master: 878ec25593c146c9bdc3700d0fc30778 (127.0.97.2:33557)
I20260929 20:59:32.587814 388 external_mini_cluster.cc:1428] Setting key b60346d7d2e66c0137bef2de92eeb0e2
I20260929 20:59:32.588667 2140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33921
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:59:32.628634 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40803
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:59:32.960070 2490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:32.960323 2490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:32.960507 2490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:32.967484 2490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:32.967610 2490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:32.967779 2490 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 20:59:32.967856 2490 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:59:32.967983 2490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:59:32.978283 2490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:32.979574 2490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:32.980975 2490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:32.993695 2496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:32.993727 2495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:32.993669 2498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:32.994331 2490 server_base.cc:1034] running on GCE node
I20260929 20:59:32.994997 2490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:32.995869 2490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:32.997083 2490 hybrid_clock.cc:648] HybridClock initialized: now 1790715572997037 us; error 57 us; skew 500 ppm
I20260929 20:59:32.997392 2490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:59:32 dist-test-slave-1dms krb5kdc[2095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:59:33.000262 2490 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:59:33.001740 2490 webserver.cc:460] Webserver started at http://127.0.97.3:39519/ using document root <none> and password file <none>
I20260929 20:59:33.002398 2490 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:33.002542 2490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:33.002858 2490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:33.005652 2490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-2/data/instance:
uuid: "2bc5dec7746f48b99039072a2834da21"
format_stamp: "Formatted at 2026-09-29 20:59:32 on dist-test-slave-1dms"
server_key: "b20cbf8bffff88c8f2b82c433a7c47ec"
server_key_iv: "0252402e99555cf989fc38b072bd65b7"
server_key_version: "encryptionkey@0"
I20260929 20:59:33.006419 2490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-2/wal/instance:
uuid: "2bc5dec7746f48b99039072a2834da21"
format_stamp: "Formatted at 2026-09-29 20:59:32 on dist-test-slave-1dms"
server_key: "b20cbf8bffff88c8f2b82c433a7c47ec"
server_key_iv: "0252402e99555cf989fc38b072bd65b7"
server_key_version: "encryptionkey@0"
I20260929 20:59:33.011250 2490 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 20:59:33.014452 2505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:33.015623 2490 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 20:59:33.015796 2490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-2/wal
uuid: "2bc5dec7746f48b99039072a2834da21"
format_stamp: "Formatted at 2026-09-29 20:59:32 on dist-test-slave-1dms"
server_key: "b20cbf8bffff88c8f2b82c433a7c47ec"
server_key_iv: "0252402e99555cf989fc38b072bd65b7"
server_key_version: "encryptionkey@0"
I20260929 20:59:33.015983 2490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:33.034941 2490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:33.038622 2490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:33.038905 2490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:33.041386 2490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:59:33.041530 2490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:33.041657 2490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:59:33.041777 2490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:59:33 dist-test-slave-1dms krb5kdc[2095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:59:33.062202 2347 heartbeater.cc:503] Master 127.0.97.62:40803 was elected leader, sending a full tablet report...
I20260929 20:59:33.067698 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:33.042264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54293 (local address 127.0.97.62:40803)
0929 20:59:33.042445 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:33.042451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:33.043895 (+ 1444us) server_negotiation.cc:407] Connection header received
0929 20:59:33.047137 (+ 3242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:33.047146 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:33.047260 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:33.047377 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:33.049918 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.050733 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.053036 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.053906 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.054066 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:33.058712 (+ 4646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:33.058721 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:33.058729 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:33.058772 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:33.063095 (+ 4323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:33.064309 (+ 1214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:33.064316 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:33.064320 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:33.064498 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:33.065121 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:33.065128 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:33.065133 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:33.065537 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:33.065736 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:33.067303 (+ 1567us) server_negotiation.cc:299] Negotiation successful
0929 20:59:33.067458 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:59:33.070135 2514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:33.042804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40803 (local address 127.0.97.3:54293)
0929 20:59:33.043685 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:33.043747 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:33.046729 (+ 2982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:33.047704 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:33.047720 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:33.048634 (+ 914us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:33.049671 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:33.049704 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.050927 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.050931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:33.052869 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:33.052883 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.054163 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.054171 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:33.054421 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:33.055241 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:59:33.055284 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:33.058425 (+ 3141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:33.063293 (+ 4868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:33.063308 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:33.063329 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:33.064082 (+ 753us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:33.064639 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:33.064644 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:33.064648 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:33.064973 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:33.065720 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:33.065731 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:33.066286 (+ 555us) client_negotiation.cc:769] Sending connection context
0929 20:59:33.066643 (+ 357us) client_negotiation.cc:240] Negotiation successful
0929 20:59:33.068137 (+ 1494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":358,"threads_started":1}
I20260929 20:59:33.091969 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:33.077250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41689 (local address 127.0.97.62:40803)
0929 20:59:33.077412 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:33.077416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:33.078211 (+ 795us) server_negotiation.cc:407] Connection header received
0929 20:59:33.078419 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:33.078424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:33.078514 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:33.078621 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:33.080030 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.081071 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.083146 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.083538 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.083670 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:33.085807 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:33.085815 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:33.085822 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:33.085861 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:33.089188 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:33.089889 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:33.089894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:33.089898 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:33.090054 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:33.090591 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:33.090598 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:33.090604 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:33.090868 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:33.090985 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:33.091374 (+ 389us) server_negotiation.cc:299] Negotiation successful
0929 20:59:33.091593 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 20:59:33.092341 2591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:33.077393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40803 (local address 127.0.97.3:41689)
0929 20:59:33.078008 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:33.078023 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:33.078275 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:33.078678 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:33.078690 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:33.079401 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:33.079861 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:33.079878 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.081218 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.081230 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:33.082991 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:33.083005 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.083730 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.083736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:33.083972 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:33.084678 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:59:33.084696 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:33.085641 (+ 945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:33.089317 (+ 3676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:33.089322 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:33.089326 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:33.089759 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:33.090226 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:33.090244 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:33.090250 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:33.090439 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:33.090963 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:33.090967 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:33.091136 (+ 169us) client_negotiation.cc:769] Sending connection context
0929 20:59:33.091298 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 20:59:33.091682 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":373,"threads_started":1}
I20260929 20:59:33.093374 2490 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46649
I20260929 20:59:33.093452 2624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46649 every 8 connection(s)
I20260929 20:59:33.094851 2490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:59:33.097282 2625 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40803
I20260929 20:59:33.097506 2625 heartbeater.cc:464] Registering TS with master...
I20260929 20:59:33.098078 2625 heartbeater.cc:511] Master 127.0.97.62:40803 requested a full tablet report, sending...
I20260929 20:59:33.099236 2139 ts_manager.cc:194] Registered new tserver with Master: 2bc5dec7746f48b99039072a2834da21 (127.0.97.3:46649)
I20260929 20:59:33.100044 2139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:54293
I20260929 20:59:33.102516 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 2490
I20260929 20:59:33.102664 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:59:33.103098 388 external_mini_cluster.cc:1428] Setting key 982695a1d5d5a2e2d892066910566dc6
I20260929 20:59:33.105785 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 20:59:33 dist-test-slave-1dms krb5kdc[2095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 20:59:33 dist-test-slave-1dms krb5kdc[2095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:59:33.149705 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:33.136156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52718 (local address 127.0.97.62:40803)
0929 20:59:33.136373 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:33.136381 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:33.136589 (+ 208us) server_negotiation.cc:407] Connection header received
0929 20:59:33.136793 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:33.136799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:33.136933 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:33.137038 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:33.138349 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.139077 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.140473 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.140962 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.141090 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:33.144702 (+ 3612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:33.144711 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:33.144715 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:33.144745 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:33.147112 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:33.147703 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:33.147709 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:33.147712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:33.147836 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:33.148206 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:33.148211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:33.148214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:33.148507 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:33.148733 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:33.149090 (+ 357us) server_negotiation.cc:299] Negotiation successful
0929 20:59:33.149330 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 20:59:33.153605 2139 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:59:33.160365 2139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:59:33.185812 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:33.172362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39190 (local address 127.0.97.1:45601)
0929 20:59:33.173290 (+ 928us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:33.173306 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:33.174147 (+ 841us) server_negotiation.cc:407] Connection header received
0929 20:59:33.174672 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:33.174685 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:33.175008 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:33.175216 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:59:33.176135 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.178149 (+ 2014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.181461 (+ 3312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.183264 (+ 1803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.183360 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:33.183592 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:33.185323 (+ 1731us) server_negotiation.cc:299] Negotiation successful
0929 20:59:33.185487 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":510,"threads_started":1}
I20260929 20:59:33.185199 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:33.172229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45601 (local address 127.0.0.1:39190)
0929 20:59:33.173980 (+ 1751us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:33.174029 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:33.174342 (+ 313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:33.175456 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:33.175463 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:33.175493 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:59:33.175917 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:33.175930 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.178357 (+ 2427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.178365 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:33.181221 (+ 2856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:33.181241 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.183812 (+ 2571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.183821 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:33.184120 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:33.184201 (+ 81us) client_negotiation.cc:769] Sending connection context
0929 20:59:33.184371 (+ 170us) client_negotiation.cc:240] Negotiation successful
0929 20:59:33.184618 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1416,"thread_start_us":2512,"threads_started":1}
I20260929 20:59:33.190258 2277 tablet_service.cc:1467] Processing CreateTablet for tablet a3a7cc047d2e44b39cc5bc6a9757c9e4 (DEFAULT_TABLE table=test-table [id=f0987c8e43af48e6a6612990b04d7d9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:59:33.191707 2277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3a7cc047d2e44b39cc5bc6a9757c9e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:33.192615 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:33.172773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46649 (local address 127.0.0.1:56760)
0929 20:59:33.174986 (+ 2213us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:33.175004 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:33.175099 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:33.186434 (+ 11335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:33.186439 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:33.186465 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:59:33.186833 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:33.186844 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.188789 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.188794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:33.190565 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:33.190577 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.191797 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.191804 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:33.191973 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:33.192019 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 20:59:33.192124 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 20:59:33.192242 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2098,"mutex_wait_us":39,"thread_start_us":710,"threads_started":1}
I20260929 20:59:33.192615 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:33.173421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56760 (local address 127.0.97.3:46649)
0929 20:59:33.185569 (+ 12148us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:33.185577 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:33.185606 (+ 29us) server_negotiation.cc:407] Connection header received
0929 20:59:33.185686 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:33.185692 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:33.185870 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:33.186458 (+ 588us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:59:33.187060 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.188621 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.190708 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.191590 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.191735 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:33.191963 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:33.192163 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 20:59:33.192313 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11962,"thread_start_us":11775,"threads_started":1}
I20260929 20:59:33.193904 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:33.175118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33557 (local address 127.0.0.1:33558)
0929 20:59:33.185153 (+ 10035us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:33.185177 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:33.185315 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:33.186089 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:33.186096 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:33.186126 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:59:33.186542 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:33.186559 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.188475 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.188480 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:33.191032 (+ 2552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:33.191047 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.193162 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.193168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:33.193332 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:33.193361 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 20:59:33.193425 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 20:59:33.193537 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9905,"mutex_wait_us":67,"thread_start_us":9773,"threads_started":1}
I20260929 20:59:33.194844 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:33.175582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33558 (local address 127.0.97.2:33557)
0929 20:59:33.184817 (+ 9235us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:33.184827 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:33.185446 (+ 619us) server_negotiation.cc:407] Connection header received
0929 20:59:33.185585 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:33.185595 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:33.185870 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:33.186735 (+ 865us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:59:33.186879 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.188337 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.191213 (+ 2876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.192589 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.194129 (+ 1540us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:33.194308 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:33.194368 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 20:59:33.194525 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9056,"thread_start_us":8941,"threads_started":1}
I20260929 20:59:33.195815 2555 tablet_service.cc:1467] Processing CreateTablet for tablet a3a7cc047d2e44b39cc5bc6a9757c9e4 (DEFAULT_TABLE table=test-table [id=f0987c8e43af48e6a6612990b04d7d9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:59:33.197111 2555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3a7cc047d2e44b39cc5bc6a9757c9e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:33.197530 2416 tablet_service.cc:1467] Processing CreateTablet for tablet a3a7cc047d2e44b39cc5bc6a9757c9e4 (DEFAULT_TABLE table=test-table [id=f0987c8e43af48e6a6612990b04d7d9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:59:33.198525 2416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3a7cc047d2e44b39cc5bc6a9757c9e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:33.207695 2644 tablet_bootstrap.cc:492] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P b44f8fae008f4acf83a4a8b7aef1a2de: Bootstrap starting.
I20260929 20:59:33.209592 2645 tablet_bootstrap.cc:492] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 2bc5dec7746f48b99039072a2834da21: Bootstrap starting.
I20260929 20:59:33.210934 2646 tablet_bootstrap.cc:492] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 878ec25593c146c9bdc3700d0fc30778: Bootstrap starting.
I20260929 20:59:33.211995 2644 tablet_bootstrap.cc:654] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P b44f8fae008f4acf83a4a8b7aef1a2de: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:33.213081 2644 log.cc:824] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P b44f8fae008f4acf83a4a8b7aef1a2de: Log is configured to *not* fsync() on all Append() calls
I20260929 20:59:33.214066 2646 tablet_bootstrap.cc:654] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 878ec25593c146c9bdc3700d0fc30778: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:33.214066 2645 tablet_bootstrap.cc:654] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 2bc5dec7746f48b99039072a2834da21: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:33.215219 2646 log.cc:824] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 878ec25593c146c9bdc3700d0fc30778: Log is configured to *not* fsync() on all Append() calls
I20260929 20:59:33.215219 2645 log.cc:824] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 2bc5dec7746f48b99039072a2834da21: Log is configured to *not* fsync() on all Append() calls
I20260929 20:59:33.215569 2644 tablet_bootstrap.cc:492] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P b44f8fae008f4acf83a4a8b7aef1a2de: No bootstrap required, opened a new log
I20260929 20:59:33.215862 2644 ts_tablet_manager.cc:1397] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P b44f8fae008f4acf83a4a8b7aef1a2de: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 20:59:33.218151 2646 tablet_bootstrap.cc:492] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 878ec25593c146c9bdc3700d0fc30778: No bootstrap required, opened a new log
I20260929 20:59:33.218393 2645 tablet_bootstrap.cc:492] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 2bc5dec7746f48b99039072a2834da21: No bootstrap required, opened a new log
I20260929 20:59:33.218417 2646 ts_tablet_manager.cc:1397] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 878ec25593c146c9bdc3700d0fc30778: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 20:59:33.218631 2645 ts_tablet_manager.cc:1397] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 2bc5dec7746f48b99039072a2834da21: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 20:59:33.223755 2644 raft_consensus.cc:348] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P b44f8fae008f4acf83a4a8b7aef1a2de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } }
I20260929 20:59:33.224102 2644 raft_consensus.cc:374] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P b44f8fae008f4acf83a4a8b7aef1a2de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:33.224198 2644 raft_consensus.cc:729] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P b44f8fae008f4acf83a4a8b7aef1a2de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b44f8fae008f4acf83a4a8b7aef1a2de, State: Initialized, Role: FOLLOWER
I20260929 20:59:33.224767 2644 consensus_queue.cc:260] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P b44f8fae008f4acf83a4a8b7aef1a2de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } }
I20260929 20:59:33.226459 2644 ts_tablet_manager.cc:1428] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P b44f8fae008f4acf83a4a8b7aef1a2de: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260929 20:59:33.229130 2646 raft_consensus.cc:348] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 878ec25593c146c9bdc3700d0fc30778 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } }
I20260929 20:59:33.229130 2645 raft_consensus.cc:348] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 2bc5dec7746f48b99039072a2834da21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } }
I20260929 20:59:33.229538 2645 raft_consensus.cc:374] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 2bc5dec7746f48b99039072a2834da21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:33.229535 2646 raft_consensus.cc:374] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 878ec25593c146c9bdc3700d0fc30778 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:33.229637 2646 raft_consensus.cc:729] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 878ec25593c146c9bdc3700d0fc30778 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 878ec25593c146c9bdc3700d0fc30778, State: Initialized, Role: FOLLOWER
I20260929 20:59:33.229637 2645 raft_consensus.cc:729] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 2bc5dec7746f48b99039072a2834da21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bc5dec7746f48b99039072a2834da21, State: Initialized, Role: FOLLOWER
I20260929 20:59:33.230218 2645 consensus_queue.cc:260] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 2bc5dec7746f48b99039072a2834da21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } }
I20260929 20:59:33.230217 2646 consensus_queue.cc:260] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 878ec25593c146c9bdc3700d0fc30778 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } }
I20260929 20:59:33.231313 2486 heartbeater.cc:503] Master 127.0.97.62:40803 was elected leader, sending a full tablet report...
I20260929 20:59:33.231796 2625 heartbeater.cc:503] Master 127.0.97.62:40803 was elected leader, sending a full tablet report...
I20260929 20:59:33.232016 2646 ts_tablet_manager.cc:1428] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 878ec25593c146c9bdc3700d0fc30778: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260929 20:59:33.232283 2645 ts_tablet_manager.cc:1428] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 2bc5dec7746f48b99039072a2834da21: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
W20260929 20:59:33.305023 2348 tablet.cc:2369] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P b44f8fae008f4acf83a4a8b7aef1a2de: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:59:33.335400 2487 tablet.cc:2369] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 878ec25593c146c9bdc3700d0fc30778: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:59:33.346766 2626 tablet.cc:2369] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 2bc5dec7746f48b99039072a2834da21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:59:33.502118 2650 raft_consensus.cc:482] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P b44f8fae008f4acf83a4a8b7aef1a2de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:59:33.502468 2650 raft_consensus.cc:504] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P b44f8fae008f4acf83a4a8b7aef1a2de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } }
I20260929 20:59:33.504071 2650 leader_election.cc:288] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P b44f8fae008f4acf83a4a8b7aef1a2de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bc5dec7746f48b99039072a2834da21 (127.0.97.3:46649), 878ec25593c146c9bdc3700d0fc30778 (127.0.97.2:33557)
I20260929 20:59:33.510739 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:33.504567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42229 (local address 127.0.97.3:46649)
0929 20:59:33.504764 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:33.504772 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:33.505323 (+ 551us) server_negotiation.cc:407] Connection header received
0929 20:59:33.505485 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:33.505490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:33.505574 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:33.505673 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:59:33.506493 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.507180 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.509058 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.509799 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.509944 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:33.510053 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:33.510332 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 20:59:33.510457 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:59:33.510767 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:33.504488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54619 (local address 127.0.97.2:33557)
0929 20:59:33.504711 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:33.504721 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:33.505232 (+ 511us) server_negotiation.cc:407] Connection header received
0929 20:59:33.505489 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:33.505495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:33.505574 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:33.505673 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:59:33.506540 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.507290 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.509131 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.509863 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.510012 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:33.510119 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:33.510377 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 20:59:33.510503 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 20:59:33.510917 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:33.504393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46649 (local address 127.0.97.1:42229)
0929 20:59:33.505207 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:33.505221 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:33.505341 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:33.505709 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:33.505715 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:33.505751 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:59:33.506351 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:33.506361 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.507336 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.507342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:33.508914 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:33.508930 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.509999 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.510005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:33.510173 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:33.510196 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 20:59:33.510307 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 20:59:33.510457 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":95,"thread_start_us":567,"threads_started":1}
I20260929 20:59:33.511013 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:33.504335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33557 (local address 127.0.97.1:54619)
0929 20:59:33.505119 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:33.505138 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:33.505307 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:33.505709 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:33.505715 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:33.505808 (+ 93us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:59:33.506309 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:33.506328 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.507438 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.507445 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:33.508986 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:33.508998 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.510080 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.510086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:33.510231 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:33.510252 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 20:59:33.510354 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 20:59:33.510499 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"mutex_wait_us":38,"thread_start_us":419,"threads_started":1}
I20260929 20:59:33.511879 2441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a7cc047d2e44b39cc5bc6a9757c9e4" candidate_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "878ec25593c146c9bdc3700d0fc30778" is_pre_election: true
I20260929 20:59:33.511884 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a7cc047d2e44b39cc5bc6a9757c9e4" candidate_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bc5dec7746f48b99039072a2834da21" is_pre_election: true
I20260929 20:59:33.512320 2441 raft_consensus.cc:2454] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 878ec25593c146c9bdc3700d0fc30778 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b44f8fae008f4acf83a4a8b7aef1a2de in term 0.
I20260929 20:59:33.512334 2575 raft_consensus.cc:2454] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 2bc5dec7746f48b99039072a2834da21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b44f8fae008f4acf83a4a8b7aef1a2de in term 0.
I20260929 20:59:33.512964 2230 leader_election.cc:302] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P b44f8fae008f4acf83a4a8b7aef1a2de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 878ec25593c146c9bdc3700d0fc30778, b44f8fae008f4acf83a4a8b7aef1a2de; no voters:
I20260929 20:59:33.513342 2650 raft_consensus.cc:2781] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P b44f8fae008f4acf83a4a8b7aef1a2de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:59:33.513485 2650 raft_consensus.cc:482] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P b44f8fae008f4acf83a4a8b7aef1a2de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:59:33.513588 2650 raft_consensus.cc:3037] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P b44f8fae008f4acf83a4a8b7aef1a2de [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:33.515640 2650 raft_consensus.cc:504] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P b44f8fae008f4acf83a4a8b7aef1a2de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } }
I20260929 20:59:33.516186 2650 leader_election.cc:288] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P b44f8fae008f4acf83a4a8b7aef1a2de [CANDIDATE]: Term 1 election: Requested vote from peers 2bc5dec7746f48b99039072a2834da21 (127.0.97.3:46649), 878ec25593c146c9bdc3700d0fc30778 (127.0.97.2:33557)
I20260929 20:59:33.516520 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a7cc047d2e44b39cc5bc6a9757c9e4" candidate_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bc5dec7746f48b99039072a2834da21"
I20260929 20:59:33.516698 2575 raft_consensus.cc:3037] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 2bc5dec7746f48b99039072a2834da21 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:33.516637 2441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a7cc047d2e44b39cc5bc6a9757c9e4" candidate_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "878ec25593c146c9bdc3700d0fc30778"
I20260929 20:59:33.516881 2441 raft_consensus.cc:3037] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 878ec25593c146c9bdc3700d0fc30778 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:33.518729 2575 raft_consensus.cc:2454] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 2bc5dec7746f48b99039072a2834da21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b44f8fae008f4acf83a4a8b7aef1a2de in term 1.
I20260929 20:59:33.518970 2441 raft_consensus.cc:2454] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 878ec25593c146c9bdc3700d0fc30778 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b44f8fae008f4acf83a4a8b7aef1a2de in term 1.
I20260929 20:59:33.519130 2227 leader_election.cc:302] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P b44f8fae008f4acf83a4a8b7aef1a2de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bc5dec7746f48b99039072a2834da21, b44f8fae008f4acf83a4a8b7aef1a2de; no voters:
I20260929 20:59:33.519431 2650 raft_consensus.cc:2781] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P b44f8fae008f4acf83a4a8b7aef1a2de [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:59:33.519869 2650 raft_consensus.cc:686] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P b44f8fae008f4acf83a4a8b7aef1a2de [term 1 LEADER]: Becoming Leader. State: Replica: b44f8fae008f4acf83a4a8b7aef1a2de, State: Running, Role: LEADER
I20260929 20:59:33.520381 2650 consensus_queue.cc:237] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P b44f8fae008f4acf83a4a8b7aef1a2de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } }
I20260929 20:59:33.524360 2140 catalog_manager.cc:5495] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P b44f8fae008f4acf83a4a8b7aef1a2de reported cstate change: term changed from 0 to 1, leader changed from <none> to b44f8fae008f4acf83a4a8b7aef1a2de (127.0.97.1). New cstate: current_term: 1 leader_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } health_report { overall_health: UNKNOWN } } }
Sep 29 20:59:33 dist-test-slave-1dms krb5kdc[2095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:59:33.584784 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:33.571327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40803 (local address 127.0.0.1:52732)
0929 20:59:33.572014 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:33.572033 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:33.572207 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:33.572516 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:33.572520 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:33.573287 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:33.573540 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:33.573549 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.574543 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.574551 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:33.575525 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:33.575537 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.576244 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.576250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:33.576425 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:33.577085 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:59:33.577104 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:33.579201 (+ 2097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:33.582413 (+ 3212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:33.582422 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:33.582427 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:33.582809 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:33.583199 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:33.583202 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:33.583205 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:33.583327 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:33.583926 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:33.583932 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:33.584074 (+ 142us) client_negotiation.cc:769] Sending connection context
0929 20:59:33.584199 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 20:59:33.584388 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":331,"threads_started":1}
I20260929 20:59:33.584939 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:33.571316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52732 (local address 127.0.97.62:40803)
0929 20:59:33.571492 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:33.571497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:33.572126 (+ 629us) server_negotiation.cc:407] Connection header received
0929 20:59:33.572325 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:33.572329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:33.572390 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:33.572489 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:33.573673 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.574415 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.575657 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.576065 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.576202 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:33.579377 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:33.579382 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:33.579385 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:33.579411 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:33.582247 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:33.582922 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:33.582928 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:33.582931 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:33.583096 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:33.583429 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:33.583436 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:33.583442 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:33.583813 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:33.583949 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:33.584331 (+ 382us) server_negotiation.cc:299] Negotiation successful
0929 20:59:33.584536 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:59:33.590379 2140 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52732:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 20:59:33.591379 2140 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:59:33.597839 2277 tablet_service.cc:1467] Processing CreateTablet for tablet e65a3966d4b0452bad2478bb601eb94b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7fd303c275264b04be67dcf28cbb92c9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:59:33.597839 2416 tablet_service.cc:1467] Processing CreateTablet for tablet e65a3966d4b0452bad2478bb601eb94b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7fd303c275264b04be67dcf28cbb92c9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:59:33.598773 2277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e65a3966d4b0452bad2478bb601eb94b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:33.598781 2416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e65a3966d4b0452bad2478bb601eb94b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:33.598300 2555 tablet_service.cc:1467] Processing CreateTablet for tablet e65a3966d4b0452bad2478bb601eb94b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7fd303c275264b04be67dcf28cbb92c9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:59:33.599146 2555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e65a3966d4b0452bad2478bb601eb94b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:33.605196 2645 tablet_bootstrap.cc:492] T e65a3966d4b0452bad2478bb601eb94b P 2bc5dec7746f48b99039072a2834da21: Bootstrap starting.
I20260929 20:59:33.605971 2646 tablet_bootstrap.cc:492] T e65a3966d4b0452bad2478bb601eb94b P 878ec25593c146c9bdc3700d0fc30778: Bootstrap starting.
I20260929 20:59:33.606977 2645 tablet_bootstrap.cc:654] T e65a3966d4b0452bad2478bb601eb94b P 2bc5dec7746f48b99039072a2834da21: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:33.607609 2646 tablet_bootstrap.cc:654] T e65a3966d4b0452bad2478bb601eb94b P 878ec25593c146c9bdc3700d0fc30778: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:33.608610 2644 tablet_bootstrap.cc:492] T e65a3966d4b0452bad2478bb601eb94b P b44f8fae008f4acf83a4a8b7aef1a2de: Bootstrap starting.
I20260929 20:59:33.609190 2645 tablet_bootstrap.cc:492] T e65a3966d4b0452bad2478bb601eb94b P 2bc5dec7746f48b99039072a2834da21: No bootstrap required, opened a new log
I20260929 20:59:33.609290 2645 ts_tablet_manager.cc:1397] T e65a3966d4b0452bad2478bb601eb94b P 2bc5dec7746f48b99039072a2834da21: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 20:59:33.609910 2644 tablet_bootstrap.cc:654] T e65a3966d4b0452bad2478bb601eb94b P b44f8fae008f4acf83a4a8b7aef1a2de: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:33.609813 2645 raft_consensus.cc:348] T e65a3966d4b0452bad2478bb601eb94b P 2bc5dec7746f48b99039072a2834da21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } }
I20260929 20:59:33.609977 2645 raft_consensus.cc:374] T e65a3966d4b0452bad2478bb601eb94b P 2bc5dec7746f48b99039072a2834da21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:33.610033 2645 raft_consensus.cc:729] T e65a3966d4b0452bad2478bb601eb94b P 2bc5dec7746f48b99039072a2834da21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bc5dec7746f48b99039072a2834da21, State: Initialized, Role: FOLLOWER
I20260929 20:59:33.610246 2645 consensus_queue.cc:260] T e65a3966d4b0452bad2478bb601eb94b P 2bc5dec7746f48b99039072a2834da21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } }
I20260929 20:59:33.610810 2645 ts_tablet_manager.cc:1428] T e65a3966d4b0452bad2478bb601eb94b P 2bc5dec7746f48b99039072a2834da21: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 20:59:33.610651 2651 tablet_replica.cc:439] T e65a3966d4b0452bad2478bb601eb94b P 2bc5dec7746f48b99039072a2834da21: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } } }
I20260929 20:59:33.610942 2651 tablet_replica.cc:442] T e65a3966d4b0452bad2478bb601eb94b P 2bc5dec7746f48b99039072a2834da21: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:59:33.611959 2646 tablet_bootstrap.cc:492] T e65a3966d4b0452bad2478bb601eb94b P 878ec25593c146c9bdc3700d0fc30778: No bootstrap required, opened a new log
I20260929 20:59:33.612073 2644 tablet_bootstrap.cc:492] T e65a3966d4b0452bad2478bb601eb94b P b44f8fae008f4acf83a4a8b7aef1a2de: No bootstrap required, opened a new log
I20260929 20:59:33.612109 2646 ts_tablet_manager.cc:1397] T e65a3966d4b0452bad2478bb601eb94b P 878ec25593c146c9bdc3700d0fc30778: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260929 20:59:33.612210 2644 ts_tablet_manager.cc:1397] T e65a3966d4b0452bad2478bb601eb94b P b44f8fae008f4acf83a4a8b7aef1a2de: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 20:59:33.612716 2646 raft_consensus.cc:348] T e65a3966d4b0452bad2478bb601eb94b P 878ec25593c146c9bdc3700d0fc30778 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } }
I20260929 20:59:33.612982 2646 raft_consensus.cc:374] T e65a3966d4b0452bad2478bb601eb94b P 878ec25593c146c9bdc3700d0fc30778 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:33.613077 2646 raft_consensus.cc:729] T e65a3966d4b0452bad2478bb601eb94b P 878ec25593c146c9bdc3700d0fc30778 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 878ec25593c146c9bdc3700d0fc30778, State: Initialized, Role: FOLLOWER
I20260929 20:59:33.613015 2644 raft_consensus.cc:348] T e65a3966d4b0452bad2478bb601eb94b P b44f8fae008f4acf83a4a8b7aef1a2de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } }
I20260929 20:59:33.613262 2644 raft_consensus.cc:374] T e65a3966d4b0452bad2478bb601eb94b P b44f8fae008f4acf83a4a8b7aef1a2de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:33.613355 2644 raft_consensus.cc:729] T e65a3966d4b0452bad2478bb601eb94b P b44f8fae008f4acf83a4a8b7aef1a2de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b44f8fae008f4acf83a4a8b7aef1a2de, State: Initialized, Role: FOLLOWER
I20260929 20:59:33.613320 2646 consensus_queue.cc:260] T e65a3966d4b0452bad2478bb601eb94b P 878ec25593c146c9bdc3700d0fc30778 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } }
I20260929 20:59:33.613574 2644 consensus_queue.cc:260] T e65a3966d4b0452bad2478bb601eb94b P b44f8fae008f4acf83a4a8b7aef1a2de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } }
I20260929 20:59:33.613981 2646 ts_tablet_manager.cc:1428] T e65a3966d4b0452bad2478bb601eb94b P 878ec25593c146c9bdc3700d0fc30778: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 20:59:33.613972 2650 tablet_replica.cc:439] T e65a3966d4b0452bad2478bb601eb94b P b44f8fae008f4acf83a4a8b7aef1a2de: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } } }
I20260929 20:59:33.613832 2652 tablet_replica.cc:439] T e65a3966d4b0452bad2478bb601eb94b P 878ec25593c146c9bdc3700d0fc30778: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } } }
I20260929 20:59:33.614207 2650 tablet_replica.cc:442] T e65a3966d4b0452bad2478bb601eb94b P b44f8fae008f4acf83a4a8b7aef1a2de: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:59:33.614235 2652 tablet_replica.cc:442] T e65a3966d4b0452bad2478bb601eb94b P 878ec25593c146c9bdc3700d0fc30778: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:59:33.615119 2644 ts_tablet_manager.cc:1428] T e65a3966d4b0452bad2478bb601eb94b P b44f8fae008f4acf83a4a8b7aef1a2de: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 20:59:33.686991 2652 raft_consensus.cc:482] T e65a3966d4b0452bad2478bb601eb94b P 878ec25593c146c9bdc3700d0fc30778 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:59:33.687263 2652 raft_consensus.cc:504] T e65a3966d4b0452bad2478bb601eb94b P 878ec25593c146c9bdc3700d0fc30778 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } }
I20260929 20:59:33.688899 2652 leader_election.cc:288] T e65a3966d4b0452bad2478bb601eb94b P 878ec25593c146c9bdc3700d0fc30778 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b44f8fae008f4acf83a4a8b7aef1a2de (127.0.97.1:45601), 2bc5dec7746f48b99039072a2834da21 (127.0.97.3:46649)
I20260929 20:59:33.696470 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:33.689429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44041 (local address 127.0.97.3:46649)
0929 20:59:33.689656 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:33.689665 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:33.690591 (+ 926us) server_negotiation.cc:407] Connection header received
0929 20:59:33.690810 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:33.690814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:33.690870 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:33.690967 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:59:33.691605 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.692328 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.694621 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.695374 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.695535 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:33.695632 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:33.696039 (+ 407us) server_negotiation.cc:299] Negotiation successful
0929 20:59:33.696185 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 20:59:33.696522 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:33.689217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49461 (local address 127.0.97.1:45601)
0929 20:59:33.690570 (+ 1353us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:33.690575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:33.690589 (+ 14us) server_negotiation.cc:407] Connection header received
0929 20:59:33.690647 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:33.690650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:33.690710 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:33.690820 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:59:33.691470 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.692426 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.694690 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.695446 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.695652 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:33.695763 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:33.696050 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 20:59:33.696178 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1248,"thread_start_us":1141,"threads_started":1}
I20260929 20:59:33.696620 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:33.689213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46649 (local address 127.0.97.2:44041)
0929 20:59:33.690464 (+ 1251us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:33.690488 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:33.690640 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:33.690981 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:33.690985 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:33.691025 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:59:33.691453 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:33.691461 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.692455 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.692463 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:33.694443 (+ 1980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:33.694455 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.695586 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.695593 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:33.695852 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:33.695878 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 20:59:33.696014 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 20:59:33.696149 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1122,"mutex_wait_us":56,"thread_start_us":455,"threads_started":1}
I20260929 20:59:33.697014 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:33.689079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45601 (local address 127.0.97.2:49461)
0929 20:59:33.690067 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:33.690096 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:33.690244 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:33.690848 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:33.690852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:33.690879 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:59:33.691299 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:33.691312 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.692598 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.692603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:33.694526 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:33.694541 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.695749 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.695755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:33.695900 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:33.695921 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 20:59:33.696026 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 20:59:33.696150 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"mutex_wait_us":136,"thread_start_us":1028,"threads_started":1}
I20260929 20:59:33.697322 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e65a3966d4b0452bad2478bb601eb94b" candidate_uuid: "878ec25593c146c9bdc3700d0fc30778" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bc5dec7746f48b99039072a2834da21" is_pre_election: true
I20260929 20:59:33.697602 2575 raft_consensus.cc:2454] T e65a3966d4b0452bad2478bb601eb94b P 2bc5dec7746f48b99039072a2834da21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 878ec25593c146c9bdc3700d0fc30778 in term 0.
I20260929 20:59:33.697960 2302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e65a3966d4b0452bad2478bb601eb94b" candidate_uuid: "878ec25593c146c9bdc3700d0fc30778" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" is_pre_election: true
I20260929 20:59:33.698100 2367 leader_election.cc:302] T e65a3966d4b0452bad2478bb601eb94b P 878ec25593c146c9bdc3700d0fc30778 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bc5dec7746f48b99039072a2834da21, 878ec25593c146c9bdc3700d0fc30778; no voters:
I20260929 20:59:33.698333 2302 raft_consensus.cc:2454] T e65a3966d4b0452bad2478bb601eb94b P b44f8fae008f4acf83a4a8b7aef1a2de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 878ec25593c146c9bdc3700d0fc30778 in term 0.
I20260929 20:59:33.698521 2652 raft_consensus.cc:2781] T e65a3966d4b0452bad2478bb601eb94b P 878ec25593c146c9bdc3700d0fc30778 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:59:33.698652 2652 raft_consensus.cc:482] T e65a3966d4b0452bad2478bb601eb94b P 878ec25593c146c9bdc3700d0fc30778 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:59:33.698764 2652 raft_consensus.cc:3037] T e65a3966d4b0452bad2478bb601eb94b P 878ec25593c146c9bdc3700d0fc30778 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:33.701228 2652 raft_consensus.cc:504] T e65a3966d4b0452bad2478bb601eb94b P 878ec25593c146c9bdc3700d0fc30778 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } }
I20260929 20:59:33.701864 2652 leader_election.cc:288] T e65a3966d4b0452bad2478bb601eb94b P 878ec25593c146c9bdc3700d0fc30778 [CANDIDATE]: Term 1 election: Requested vote from peers b44f8fae008f4acf83a4a8b7aef1a2de (127.0.97.1:45601), 2bc5dec7746f48b99039072a2834da21 (127.0.97.3:46649)
I20260929 20:59:33.702346 2302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e65a3966d4b0452bad2478bb601eb94b" candidate_uuid: "878ec25593c146c9bdc3700d0fc30778" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de"
I20260929 20:59:33.702330 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e65a3966d4b0452bad2478bb601eb94b" candidate_uuid: "878ec25593c146c9bdc3700d0fc30778" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bc5dec7746f48b99039072a2834da21"
I20260929 20:59:33.702527 2302 raft_consensus.cc:3037] T e65a3966d4b0452bad2478bb601eb94b P b44f8fae008f4acf83a4a8b7aef1a2de [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:33.702545 2575 raft_consensus.cc:3037] T e65a3966d4b0452bad2478bb601eb94b P 2bc5dec7746f48b99039072a2834da21 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:33.705669 2575 raft_consensus.cc:2454] T e65a3966d4b0452bad2478bb601eb94b P 2bc5dec7746f48b99039072a2834da21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 878ec25593c146c9bdc3700d0fc30778 in term 1.
I20260929 20:59:33.705693 2302 raft_consensus.cc:2454] T e65a3966d4b0452bad2478bb601eb94b P b44f8fae008f4acf83a4a8b7aef1a2de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 878ec25593c146c9bdc3700d0fc30778 in term 1.
I20260929 20:59:33.706063 2367 leader_election.cc:302] T e65a3966d4b0452bad2478bb601eb94b P 878ec25593c146c9bdc3700d0fc30778 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bc5dec7746f48b99039072a2834da21, 878ec25593c146c9bdc3700d0fc30778; no voters:
I20260929 20:59:33.706336 2652 raft_consensus.cc:2781] T e65a3966d4b0452bad2478bb601eb94b P 878ec25593c146c9bdc3700d0fc30778 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:59:33.706770 2652 raft_consensus.cc:686] T e65a3966d4b0452bad2478bb601eb94b P 878ec25593c146c9bdc3700d0fc30778 [term 1 LEADER]: Becoming Leader. State: Replica: 878ec25593c146c9bdc3700d0fc30778, State: Running, Role: LEADER
I20260929 20:59:33.707281 2652 consensus_queue.cc:237] T e65a3966d4b0452bad2478bb601eb94b P 878ec25593c146c9bdc3700d0fc30778 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } }
I20260929 20:59:33.710675 2669 tablet_replica.cc:439] T e65a3966d4b0452bad2478bb601eb94b P 878ec25593c146c9bdc3700d0fc30778: TxnStatusTablet state changed. Reason: New leader 878ec25593c146c9bdc3700d0fc30778. Latest consensus state: current_term: 1 leader_uuid: "878ec25593c146c9bdc3700d0fc30778" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } } }
I20260929 20:59:33.711097 2669 tablet_replica.cc:442] T e65a3966d4b0452bad2478bb601eb94b P 878ec25593c146c9bdc3700d0fc30778: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:59:33.711786 2671 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 20:59:33.712024 2671 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 20:59:33.711951 2140 catalog_manager.cc:5495] T e65a3966d4b0452bad2478bb601eb94b P 878ec25593c146c9bdc3700d0fc30778 reported cstate change: term changed from 0 to 1, leader changed from <none> to 878ec25593c146c9bdc3700d0fc30778 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "878ec25593c146c9bdc3700d0fc30778" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } health_report { overall_health: UNKNOWN } } }
I20260929 20:59:33.737507 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:33.732126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33557 (local address 127.0.0.1:33562)
0929 20:59:33.732374 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:33.732395 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:33.732526 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:33.732978 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:33.732983 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:33.732994 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:59:33.733196 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:33.733205 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.734198 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.734206 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:33.735279 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:33.735290 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.736014 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.736020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:33.736157 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:33.736205 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:59:33.736993 (+ 788us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:59:33.737010 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 20:59:33.737091 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 20:59:33.737228 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 20:59:33.737535 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:33.732315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33562 (local address 127.0.97.2:33557)
0929 20:59:33.732487 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:33.732493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:33.732511 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:59:33.732715 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:33.732726 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:33.732799 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:33.732967 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:59:33.733322 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.734082 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.735409 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.735860 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.735994 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:33.736380 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:59:33.736880 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:59:33.737031 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:33.737105 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 20:59:33.737184 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 20:59:33.744932 2302 raft_consensus.cc:1261] T e65a3966d4b0452bad2478bb601eb94b P b44f8fae008f4acf83a4a8b7aef1a2de [term 1 FOLLOWER]: Refusing update from remote peer 878ec25593c146c9bdc3700d0fc30778: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:59:33.745762 2669 consensus_queue.cc:1035] T e65a3966d4b0452bad2478bb601eb94b P 878ec25593c146c9bdc3700d0fc30778 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:59:33.745867 2575 raft_consensus.cc:1261] T e65a3966d4b0452bad2478bb601eb94b P 2bc5dec7746f48b99039072a2834da21 [term 1 FOLLOWER]: Refusing update from remote peer 878ec25593c146c9bdc3700d0fc30778: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:59:33.746610 2669 consensus_queue.cc:1035] T e65a3966d4b0452bad2478bb601eb94b P 878ec25593c146c9bdc3700d0fc30778 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:59:33.753628 2650 tablet_replica.cc:439] T e65a3966d4b0452bad2478bb601eb94b P b44f8fae008f4acf83a4a8b7aef1a2de: TxnStatusTablet state changed. Reason: New leader 878ec25593c146c9bdc3700d0fc30778. Latest consensus state: current_term: 1 leader_uuid: "878ec25593c146c9bdc3700d0fc30778" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } } }
I20260929 20:59:33.753953 2650 tablet_replica.cc:442] T e65a3966d4b0452bad2478bb601eb94b P b44f8fae008f4acf83a4a8b7aef1a2de: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:59:33.757653 2675 mvcc.cc:205] Tried to move back new op lower bound from 7334770990050492416 to 7334770989908824064. Current Snapshot: MvccSnapshot[applied={T|T < 7334770990050492416}]
I20260929 20:59:33.758677 2650 tablet_replica.cc:439] T e65a3966d4b0452bad2478bb601eb94b P b44f8fae008f4acf83a4a8b7aef1a2de: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "878ec25593c146c9bdc3700d0fc30778" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } } }
I20260929 20:59:33.759027 2650 tablet_replica.cc:442] T e65a3966d4b0452bad2478bb601eb94b P b44f8fae008f4acf83a4a8b7aef1a2de: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:59:33.760643 2674 mvcc.cc:205] Tried to move back new op lower bound from 7334770990050492416 to 7334770989908824064. Current Snapshot: MvccSnapshot[applied={T|T < 7334770990050492416}]
I20260929 20:59:33.761300 2652 tablet_replica.cc:439] T e65a3966d4b0452bad2478bb601eb94b P 878ec25593c146c9bdc3700d0fc30778: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "878ec25593c146c9bdc3700d0fc30778" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } } }
I20260929 20:59:33.761878 2652 tablet_replica.cc:442] T e65a3966d4b0452bad2478bb601eb94b P 878ec25593c146c9bdc3700d0fc30778: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:59:33.770673 2651 tablet_replica.cc:439] T e65a3966d4b0452bad2478bb601eb94b P 2bc5dec7746f48b99039072a2834da21: TxnStatusTablet state changed. Reason: New leader 878ec25593c146c9bdc3700d0fc30778. Latest consensus state: current_term: 1 leader_uuid: "878ec25593c146c9bdc3700d0fc30778" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } } }
I20260929 20:59:33.770955 2651 tablet_replica.cc:442] T e65a3966d4b0452bad2478bb601eb94b P 2bc5dec7746f48b99039072a2834da21: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:59:33.771865 2669 tablet_replica.cc:439] T e65a3966d4b0452bad2478bb601eb94b P 878ec25593c146c9bdc3700d0fc30778: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "878ec25593c146c9bdc3700d0fc30778" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } } }
I20260929 20:59:33.772130 2669 tablet_replica.cc:442] T e65a3966d4b0452bad2478bb601eb94b P 878ec25593c146c9bdc3700d0fc30778: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:59:33.774963 2677 mvcc.cc:205] Tried to move back new op lower bound from 7334770990050492416 to 7334770989908824064. Current Snapshot: MvccSnapshot[applied={T|T < 7334770990050492416}]
I20260929 20:59:33.777565 2651 tablet_replica.cc:439] T e65a3966d4b0452bad2478bb601eb94b P 2bc5dec7746f48b99039072a2834da21: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "878ec25593c146c9bdc3700d0fc30778" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } } }
I20260929 20:59:33.777782 2651 tablet_replica.cc:442] T e65a3966d4b0452bad2478bb601eb94b P 2bc5dec7746f48b99039072a2834da21: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:59:33.780033 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:33.770272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39202 (local address 127.0.97.1:45601)
0929 20:59:33.770456 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:33.770461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:33.771351 (+ 890us) server_negotiation.cc:407] Connection header received
0929 20:59:33.771669 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:33.771675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:33.771773 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:33.771894 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:59:33.772764 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.773868 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.776538 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.777203 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.777329 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:33.778687 (+ 1358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:59:33.779375 (+ 688us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:59:33.779470 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:33.779677 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 20:59:33.779762 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 20:59:33.795256 2687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:33.789494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33557 (local address 127.0.97.1:36817)
0929 20:59:33.790004 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:33.790023 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:33.790143 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:33.790510 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:33.790514 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:33.790524 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:59:33.790791 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:33.790798 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.792154 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.792157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:33.793306 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:33.793317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.794256 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.794260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:33.794365 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:33.794415 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:59:33.794811 (+ 396us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:59:33.794818 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 20:59:33.794873 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 20:59:33.794961 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":261,"threads_started":1}
I20260929 20:59:33.795264 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:33.789613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36817 (local address 127.0.97.2:33557)
0929 20:59:33.789791 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:33.789797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:33.790097 (+ 300us) server_negotiation.cc:407] Connection header received
0929 20:59:33.790254 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:33.790261 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:33.790369 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:33.790465 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:59:33.790928 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.792044 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.793428 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.794094 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.794223 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:33.794592 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:59:33.794718 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:59:33.794798 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:33.794914 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 20:59:33.794986 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:59:33.805155 2575 raft_consensus.cc:1261] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 2bc5dec7746f48b99039072a2834da21 [term 1 FOLLOWER]: Refusing update from remote peer b44f8fae008f4acf83a4a8b7aef1a2de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:59:33.805316 2441 raft_consensus.cc:1261] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 878ec25593c146c9bdc3700d0fc30778 [term 1 FOLLOWER]: Refusing update from remote peer b44f8fae008f4acf83a4a8b7aef1a2de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:59:33.805795 2655 consensus_queue.cc:1035] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P b44f8fae008f4acf83a4a8b7aef1a2de [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:59:33.806208 2650 consensus_queue.cc:1035] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P b44f8fae008f4acf83a4a8b7aef1a2de [LEADER]: Connected to new peer: Peer: permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:59:33.845201 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:33.838720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33519 (local address 127.0.97.1:45601)
0929 20:59:33.838920 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:33.838929 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:33.839549 (+ 620us) server_negotiation.cc:407] Connection header received
0929 20:59:33.839833 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:33.839838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:33.839913 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:33.840000 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:59:33.840657 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.841462 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.843561 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.843976 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.844077 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:33.844568 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:59:33.844662 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:59:33.844730 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:33.844916 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 20:59:33.844972 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 20:59:33.845441 2692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:33.838694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45601 (local address 127.0.97.2:33519)
0929 20:59:33.839455 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:33.839473 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:33.839643 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:33.840128 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:33.840133 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:33.840147 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:59:33.840517 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:33.840529 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.841595 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.841601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:33.843392 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:33.843406 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:33.844157 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:33.844166 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:33.844344 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:33.844415 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:59:33.844752 (+ 337us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:59:33.844777 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 20:59:33.844888 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 20:59:33.845025 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":420,"threads_started":1}
I20260929 20:59:33.896569 2140 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:59:33.896894 2140 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:59:33.900770 2140 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d36341f2e2ba4bccbbae2e9c386e6696: Sending DeleteTablet for 3 replicas of tablet a3a7cc047d2e44b39cc5bc6a9757c9e4
I20260929 20:59:33.901859 2277 tablet_service.cc:1514] Processing DeleteTablet for tablet a3a7cc047d2e44b39cc5bc6a9757c9e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:59:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39190
I20260929 20:59:33.901952 2555 tablet_service.cc:1514] Processing DeleteTablet for tablet a3a7cc047d2e44b39cc5bc6a9757c9e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:59:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56760
I20260929 20:59:33.903267 2416 tablet_service.cc:1514] Processing DeleteTablet for tablet a3a7cc047d2e44b39cc5bc6a9757c9e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:59:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33558
I20260929 20:59:33.904292 2139 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:59:33.904987 2139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:59:33.912321 2554 tablet_service.cc:1467] Processing CreateTablet for tablet 6965e026f5ef48138c76d87034f8aa96 (DEFAULT_TABLE table=test-table [id=75d51869dbe54bc0a75ee366cde53121]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:59:33.913324 2695 tablet_replica.cc:330] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P b44f8fae008f4acf83a4a8b7aef1a2de: stopping tablet replica
I20260929 20:59:33.913755 2415 tablet_service.cc:1467] Processing CreateTablet for tablet 6965e026f5ef48138c76d87034f8aa96 (DEFAULT_TABLE table=test-table [id=75d51869dbe54bc0a75ee366cde53121]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:59:33.913754 2276 tablet_service.cc:1467] Processing CreateTablet for tablet 6965e026f5ef48138c76d87034f8aa96 (DEFAULT_TABLE table=test-table [id=75d51869dbe54bc0a75ee366cde53121]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:59:33.914186 2695 raft_consensus.cc:2229] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P b44f8fae008f4acf83a4a8b7aef1a2de [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:59:33.914320 2415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6965e026f5ef48138c76d87034f8aa96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:33.914320 2276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6965e026f5ef48138c76d87034f8aa96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:33.914444 2554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6965e026f5ef48138c76d87034f8aa96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:33.921304 2646 tablet_bootstrap.cc:492] T 6965e026f5ef48138c76d87034f8aa96 P 878ec25593c146c9bdc3700d0fc30778: Bootstrap starting.
I20260929 20:59:33.922286 2644 tablet_bootstrap.cc:492] T 6965e026f5ef48138c76d87034f8aa96 P b44f8fae008f4acf83a4a8b7aef1a2de: Bootstrap starting.
I20260929 20:59:33.923012 2646 tablet_bootstrap.cc:654] T 6965e026f5ef48138c76d87034f8aa96 P 878ec25593c146c9bdc3700d0fc30778: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:33.923947 2644 tablet_bootstrap.cc:654] T 6965e026f5ef48138c76d87034f8aa96 P b44f8fae008f4acf83a4a8b7aef1a2de: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:33.928972 2646 tablet_bootstrap.cc:492] T 6965e026f5ef48138c76d87034f8aa96 P 878ec25593c146c9bdc3700d0fc30778: No bootstrap required, opened a new log
I20260929 20:59:33.929181 2646 ts_tablet_manager.cc:1397] T 6965e026f5ef48138c76d87034f8aa96 P 878ec25593c146c9bdc3700d0fc30778: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 20:59:33.929781 2697 tablet_replica.cc:330] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 878ec25593c146c9bdc3700d0fc30778: stopping tablet replica
I20260929 20:59:33.929028 2695 raft_consensus.cc:2258] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P b44f8fae008f4acf83a4a8b7aef1a2de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:59:33.930433 2645 tablet_bootstrap.cc:492] T 6965e026f5ef48138c76d87034f8aa96 P 2bc5dec7746f48b99039072a2834da21: Bootstrap starting.
I20260929 20:59:33.930154 2646 raft_consensus.cc:348] T 6965e026f5ef48138c76d87034f8aa96 P 878ec25593c146c9bdc3700d0fc30778 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } }
I20260929 20:59:33.930914 2646 raft_consensus.cc:374] T 6965e026f5ef48138c76d87034f8aa96 P 878ec25593c146c9bdc3700d0fc30778 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:33.931000 2646 raft_consensus.cc:729] T 6965e026f5ef48138c76d87034f8aa96 P 878ec25593c146c9bdc3700d0fc30778 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 878ec25593c146c9bdc3700d0fc30778, State: Initialized, Role: FOLLOWER
I20260929 20:59:33.931257 2646 consensus_queue.cc:260] T 6965e026f5ef48138c76d87034f8aa96 P 878ec25593c146c9bdc3700d0fc30778 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } }
I20260929 20:59:33.931857 2646 ts_tablet_manager.cc:1428] T 6965e026f5ef48138c76d87034f8aa96 P 878ec25593c146c9bdc3700d0fc30778: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 20:59:33.932053 2645 tablet_bootstrap.cc:654] T 6965e026f5ef48138c76d87034f8aa96 P 2bc5dec7746f48b99039072a2834da21: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:33.932555 2697 raft_consensus.cc:2229] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 878ec25593c146c9bdc3700d0fc30778 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:59:33.933990 2697 raft_consensus.cc:2258] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 878ec25593c146c9bdc3700d0fc30778 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:59:33.935199 2695 ts_tablet_manager.cc:1905] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P b44f8fae008f4acf83a4a8b7aef1a2de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:59:33.937350 2697 ts_tablet_manager.cc:1905] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 878ec25593c146c9bdc3700d0fc30778: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:59:33.947981 2695 ts_tablet_manager.cc:1918] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P b44f8fae008f4acf83a4a8b7aef1a2de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:59:33.956050 2645 tablet_bootstrap.cc:492] T 6965e026f5ef48138c76d87034f8aa96 P 2bc5dec7746f48b99039072a2834da21: No bootstrap required, opened a new log
I20260929 20:59:33.949393 2644 tablet_bootstrap.cc:492] T 6965e026f5ef48138c76d87034f8aa96 P b44f8fae008f4acf83a4a8b7aef1a2de: No bootstrap required, opened a new log
I20260929 20:59:33.956230 2645 ts_tablet_manager.cc:1397] T 6965e026f5ef48138c76d87034f8aa96 P 2bc5dec7746f48b99039072a2834da21: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.005s
I20260929 20:59:33.956382 2644 ts_tablet_manager.cc:1397] T 6965e026f5ef48138c76d87034f8aa96 P b44f8fae008f4acf83a4a8b7aef1a2de: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.005s
I20260929 20:59:33.957018 2645 raft_consensus.cc:348] T 6965e026f5ef48138c76d87034f8aa96 P 2bc5dec7746f48b99039072a2834da21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } }
I20260929 20:59:33.957260 2645 raft_consensus.cc:374] T 6965e026f5ef48138c76d87034f8aa96 P 2bc5dec7746f48b99039072a2834da21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:33.957327 2645 raft_consensus.cc:729] T 6965e026f5ef48138c76d87034f8aa96 P 2bc5dec7746f48b99039072a2834da21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bc5dec7746f48b99039072a2834da21, State: Initialized, Role: FOLLOWER
I20260929 20:59:33.957296 2644 raft_consensus.cc:348] T 6965e026f5ef48138c76d87034f8aa96 P b44f8fae008f4acf83a4a8b7aef1a2de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } }
I20260929 20:59:33.957520 2644 raft_consensus.cc:374] T 6965e026f5ef48138c76d87034f8aa96 P b44f8fae008f4acf83a4a8b7aef1a2de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:33.957595 2644 raft_consensus.cc:729] T 6965e026f5ef48138c76d87034f8aa96 P b44f8fae008f4acf83a4a8b7aef1a2de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b44f8fae008f4acf83a4a8b7aef1a2de, State: Initialized, Role: FOLLOWER
I20260929 20:59:33.957593 2645 consensus_queue.cc:260] T 6965e026f5ef48138c76d87034f8aa96 P 2bc5dec7746f48b99039072a2834da21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } }
I20260929 20:59:33.957811 2644 consensus_queue.cc:260] T 6965e026f5ef48138c76d87034f8aa96 P b44f8fae008f4acf83a4a8b7aef1a2de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } }
I20260929 20:59:33.958421 2644 ts_tablet_manager.cc:1428] T 6965e026f5ef48138c76d87034f8aa96 P b44f8fae008f4acf83a4a8b7aef1a2de: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 20:59:33.960176 2695 log.cc:1196] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P b44f8fae008f4acf83a4a8b7aef1a2de: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-0/wal/wals/a3a7cc047d2e44b39cc5bc6a9757c9e4
I20260929 20:59:33.960541 2697 ts_tablet_manager.cc:1918] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 878ec25593c146c9bdc3700d0fc30778: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:59:33.961298 2695 ts_tablet_manager.cc:1939] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P b44f8fae008f4acf83a4a8b7aef1a2de: Deleting consensus metadata
I20260929 20:59:33.961505 2697 log.cc:1196] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 878ec25593c146c9bdc3700d0fc30778: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-1/wal/wals/a3a7cc047d2e44b39cc5bc6a9757c9e4
I20260929 20:59:33.962108 2697 ts_tablet_manager.cc:1939] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 878ec25593c146c9bdc3700d0fc30778: Deleting consensus metadata
I20260929 20:59:33.962237 2645 ts_tablet_manager.cc:1428] T 6965e026f5ef48138c76d87034f8aa96 P 2bc5dec7746f48b99039072a2834da21: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260929 20:59:33.963930 2127 catalog_manager.cc:4841] TS b44f8fae008f4acf83a4a8b7aef1a2de (127.0.97.1:45601): tablet a3a7cc047d2e44b39cc5bc6a9757c9e4 (table test-table [id=f0987c8e43af48e6a6612990b04d7d9d]) successfully deleted
I20260929 20:59:33.965523 2696 tablet_replica.cc:330] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 2bc5dec7746f48b99039072a2834da21: stopping tablet replica
I20260929 20:59:33.965894 2127 catalog_manager.cc:4841] TS 878ec25593c146c9bdc3700d0fc30778 (127.0.97.2:33557): tablet a3a7cc047d2e44b39cc5bc6a9757c9e4 (table test-table [id=f0987c8e43af48e6a6612990b04d7d9d]) successfully deleted
I20260929 20:59:33.966074 2696 raft_consensus.cc:2229] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 2bc5dec7746f48b99039072a2834da21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:59:33.966480 2696 raft_consensus.cc:2258] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 2bc5dec7746f48b99039072a2834da21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:59:33.968972 2696 ts_tablet_manager.cc:1905] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 2bc5dec7746f48b99039072a2834da21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:59:33.976310 2696 ts_tablet_manager.cc:1918] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 2bc5dec7746f48b99039072a2834da21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:59:33.976457 2696 log.cc:1196] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 2bc5dec7746f48b99039072a2834da21: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-2/wal/wals/a3a7cc047d2e44b39cc5bc6a9757c9e4
I20260929 20:59:33.976958 2696 ts_tablet_manager.cc:1939] T a3a7cc047d2e44b39cc5bc6a9757c9e4 P 2bc5dec7746f48b99039072a2834da21: Deleting consensus metadata
I20260929 20:59:33.978807 2124 catalog_manager.cc:4841] TS 2bc5dec7746f48b99039072a2834da21 (127.0.97.3:46649): tablet a3a7cc047d2e44b39cc5bc6a9757c9e4 (table test-table [id=f0987c8e43af48e6a6612990b04d7d9d]) successfully deleted
I20260929 20:59:34.059067 2651 raft_consensus.cc:482] T 6965e026f5ef48138c76d87034f8aa96 P 2bc5dec7746f48b99039072a2834da21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:59:34.059296 2651 raft_consensus.cc:504] T 6965e026f5ef48138c76d87034f8aa96 P 2bc5dec7746f48b99039072a2834da21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } }
I20260929 20:59:34.061916 2651 leader_election.cc:288] T 6965e026f5ef48138c76d87034f8aa96 P 2bc5dec7746f48b99039072a2834da21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b44f8fae008f4acf83a4a8b7aef1a2de (127.0.97.1:45601), 878ec25593c146c9bdc3700d0fc30778 (127.0.97.2:33557)
I20260929 20:59:34.077673 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:34.061490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43301 (local address 127.0.97.1:45601)
0929 20:59:34.061670 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:34.061676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:34.069420 (+ 7744us) server_negotiation.cc:407] Connection header received
0929 20:59:34.070102 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:34.070108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:34.070178 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:34.070294 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:59:34.071073 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:34.072283 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:34.075131 (+ 2848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:34.076479 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:34.076571 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:34.076668 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:34.077240 (+ 572us) server_negotiation.cc:299] Negotiation successful
0929 20:59:34.077375 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 20:59:34.078521 2701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:34.061156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45601 (local address 127.0.97.3:43301)
0929 20:59:34.069259 (+ 8103us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:34.069284 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:34.069441 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:34.070521 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:34.070526 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:34.070562 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:59:34.070888 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:34.070900 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:34.072511 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:34.072517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:34.074688 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:34.074703 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:34.076821 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:34.076827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:34.077040 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:34.077079 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 20:59:34.078035 (+ 956us) client_negotiation.cc:240] Negotiation successful
0929 20:59:34.078176 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7960,"thread_start_us":7835,"threads_started":1}
I20260929 20:59:34.089054 2302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6965e026f5ef48138c76d87034f8aa96" candidate_uuid: "2bc5dec7746f48b99039072a2834da21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" is_pre_election: true
I20260929 20:59:34.089335 2302 raft_consensus.cc:2454] T 6965e026f5ef48138c76d87034f8aa96 P b44f8fae008f4acf83a4a8b7aef1a2de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bc5dec7746f48b99039072a2834da21 in term 0.
I20260929 20:59:34.089994 2509 leader_election.cc:302] T 6965e026f5ef48138c76d87034f8aa96 P 2bc5dec7746f48b99039072a2834da21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bc5dec7746f48b99039072a2834da21, b44f8fae008f4acf83a4a8b7aef1a2de; no voters:
I20260929 20:59:34.090490 2651 raft_consensus.cc:2781] T 6965e026f5ef48138c76d87034f8aa96 P 2bc5dec7746f48b99039072a2834da21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:59:34.090611 2651 raft_consensus.cc:482] T 6965e026f5ef48138c76d87034f8aa96 P 2bc5dec7746f48b99039072a2834da21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:59:34.090723 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:34.069782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39183 (local address 127.0.97.2:33557)
0929 20:59:34.069958 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:34.069964 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:34.079139 (+ 9175us) server_negotiation.cc:407] Connection header received
0929 20:59:34.079375 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:34.079380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:34.079452 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:34.079525 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:59:34.080048 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:34.081202 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:34.086709 (+ 5507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:34.088107 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:34.088202 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:34.088311 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:34.090277 (+ 1966us) server_negotiation.cc:299] Negotiation successful
0929 20:59:34.090425 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:59:34.090705 2651 raft_consensus.cc:3037] T 6965e026f5ef48138c76d87034f8aa96 P 2bc5dec7746f48b99039072a2834da21 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:34.091424 2701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:34.074930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33557 (local address 127.0.97.3:39183)
0929 20:59:34.079052 (+ 4122us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:34.079071 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:34.079238 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:34.079685 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:34.079690 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:34.079713 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:59:34.079858 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:34.079866 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:34.082892 (+ 3026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:34.082897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:34.084977 (+ 2080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:34.084991 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:34.089869 (+ 4878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:34.089875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:34.090068 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:34.090099 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 20:59:34.090949 (+ 850us) client_negotiation.cc:240] Negotiation successful
0929 20:59:34.091144 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4026,"thread_start_us":13410,"threads_started":1}
I20260929 20:59:34.092116 2441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6965e026f5ef48138c76d87034f8aa96" candidate_uuid: "2bc5dec7746f48b99039072a2834da21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "878ec25593c146c9bdc3700d0fc30778" is_pre_election: true
I20260929 20:59:34.092370 2441 raft_consensus.cc:2454] T 6965e026f5ef48138c76d87034f8aa96 P 878ec25593c146c9bdc3700d0fc30778 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bc5dec7746f48b99039072a2834da21 in term 0.
I20260929 20:59:34.094003 2651 raft_consensus.cc:504] T 6965e026f5ef48138c76d87034f8aa96 P 2bc5dec7746f48b99039072a2834da21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } }
I20260929 20:59:34.094679 2651 leader_election.cc:288] T 6965e026f5ef48138c76d87034f8aa96 P 2bc5dec7746f48b99039072a2834da21 [CANDIDATE]: Term 1 election: Requested vote from peers b44f8fae008f4acf83a4a8b7aef1a2de (127.0.97.1:45601), 878ec25593c146c9bdc3700d0fc30778 (127.0.97.2:33557)
I20260929 20:59:34.095300 2302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6965e026f5ef48138c76d87034f8aa96" candidate_uuid: "2bc5dec7746f48b99039072a2834da21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de"
I20260929 20:59:34.095510 2302 raft_consensus.cc:3037] T 6965e026f5ef48138c76d87034f8aa96 P b44f8fae008f4acf83a4a8b7aef1a2de [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:34.098562 2302 raft_consensus.cc:2454] T 6965e026f5ef48138c76d87034f8aa96 P b44f8fae008f4acf83a4a8b7aef1a2de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bc5dec7746f48b99039072a2834da21 in term 1.
I20260929 20:59:34.099756 2441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6965e026f5ef48138c76d87034f8aa96" candidate_uuid: "2bc5dec7746f48b99039072a2834da21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "878ec25593c146c9bdc3700d0fc30778"
I20260929 20:59:34.099979 2441 raft_consensus.cc:3037] T 6965e026f5ef48138c76d87034f8aa96 P 878ec25593c146c9bdc3700d0fc30778 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:34.103327 2441 raft_consensus.cc:2454] T 6965e026f5ef48138c76d87034f8aa96 P 878ec25593c146c9bdc3700d0fc30778 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bc5dec7746f48b99039072a2834da21 in term 1.
I20260929 20:59:34.103819 2509 leader_election.cc:302] T 6965e026f5ef48138c76d87034f8aa96 P 2bc5dec7746f48b99039072a2834da21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bc5dec7746f48b99039072a2834da21, 878ec25593c146c9bdc3700d0fc30778; no voters:
I20260929 20:59:34.104108 2651 raft_consensus.cc:2781] T 6965e026f5ef48138c76d87034f8aa96 P 2bc5dec7746f48b99039072a2834da21 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:59:34.117071 2651 raft_consensus.cc:686] T 6965e026f5ef48138c76d87034f8aa96 P 2bc5dec7746f48b99039072a2834da21 [term 1 LEADER]: Becoming Leader. State: Replica: 2bc5dec7746f48b99039072a2834da21, State: Running, Role: LEADER
I20260929 20:59:34.117523 2651 consensus_queue.cc:237] T 6965e026f5ef48138c76d87034f8aa96 P 2bc5dec7746f48b99039072a2834da21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } }
I20260929 20:59:34.131201 2140 catalog_manager.cc:5495] T 6965e026f5ef48138c76d87034f8aa96 P 2bc5dec7746f48b99039072a2834da21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2bc5dec7746f48b99039072a2834da21 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "2bc5dec7746f48b99039072a2834da21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } health_report { overall_health: HEALTHY } } }
I20260929 20:59:34.188338 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:34.180998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56764 (local address 127.0.97.3:46649)
0929 20:59:34.181164 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:34.181171 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:34.181404 (+ 233us) server_negotiation.cc:407] Connection header received
0929 20:59:34.181625 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:34.181631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:34.181752 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:34.181917 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:59:34.182274 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:34.183429 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:34.185369 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:34.186062 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:34.186185 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:34.186746 (+ 561us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:59:34.187525 (+ 779us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:59:34.187675 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:34.187807 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 20:59:34.187891 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 20:59:34.204705 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:34.199475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54353 (local address 127.0.97.2:33557)
0929 20:59:34.199619 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:34.199623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:34.200107 (+ 484us) server_negotiation.cc:407] Connection header received
0929 20:59:34.200309 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:34.200314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:34.200387 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:34.200474 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:59:34.201040 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:34.201725 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:34.203017 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:34.203473 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:34.203606 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:34.204011 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:59:34.204116 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:59:34.204206 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:34.204334 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 20:59:34.204404 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 20:59:34.204780 2707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:34.199384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33557 (local address 127.0.97.3:54353)
0929 20:59:34.200008 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:34.200024 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:34.200186 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:34.200496 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:34.200500 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:34.200511 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:59:34.200874 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:34.200885 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:34.201853 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:34.201860 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:34.202886 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:34.202897 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:34.203638 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:34.203645 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:34.203791 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:34.203842 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:59:34.204206 (+ 364us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:59:34.204218 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 20:59:34.204313 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 20:59:34.204438 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":342,"threads_started":1}
I20260929 20:59:34.213160 2302 raft_consensus.cc:1261] T 6965e026f5ef48138c76d87034f8aa96 P b44f8fae008f4acf83a4a8b7aef1a2de [term 1 FOLLOWER]: Refusing update from remote peer 2bc5dec7746f48b99039072a2834da21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:59:34.213361 2441 raft_consensus.cc:1261] T 6965e026f5ef48138c76d87034f8aa96 P 878ec25593c146c9bdc3700d0fc30778 [term 1 FOLLOWER]: Refusing update from remote peer 2bc5dec7746f48b99039072a2834da21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:59:34.213804 2703 consensus_queue.cc:1035] T 6965e026f5ef48138c76d87034f8aa96 P 2bc5dec7746f48b99039072a2834da21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:59:34.214178 2651 consensus_queue.cc:1035] T 6965e026f5ef48138c76d87034f8aa96 P 2bc5dec7746f48b99039072a2834da21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:59:34.258518 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:34.253233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39993 (local address 127.0.97.3:46649)
0929 20:59:34.253377 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:34.253385 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:34.253404 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:59:34.253459 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:34.253464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:34.253543 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:34.253623 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:59:34.254112 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:34.255216 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:34.256514 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:34.257242 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:34.257373 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:34.257796 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:59:34.257907 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:59:34.258010 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:34.258151 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 20:59:34.258237 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 20:59:34.258559 2692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:34.253013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46649 (local address 127.0.97.2:39993)
0929 20:59:34.253205 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:34.253225 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:34.253335 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:34.253792 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:34.253796 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:34.253811 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:59:34.253979 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:34.253988 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:34.255340 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:34.255343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:34.256377 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:34.256388 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:34.257394 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:34.257401 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:34.257608 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:34.257644 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:59:34.258016 (+ 372us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:59:34.258028 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 20:59:34.258117 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 20:59:34.258263 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 20:59:34.259827 2677 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 20:59:35.860649 2140 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:59:35.860903 2140 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:59:35.864598 2140 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d36341f2e2ba4bccbbae2e9c386e6696: Sending DeleteTablet for 3 replicas of tablet 6965e026f5ef48138c76d87034f8aa96
I20260929 20:59:35.865391 2276 tablet_service.cc:1514] Processing DeleteTablet for tablet 6965e026f5ef48138c76d87034f8aa96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:59:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39190
I20260929 20:59:35.865482 2555 tablet_service.cc:1514] Processing DeleteTablet for tablet 6965e026f5ef48138c76d87034f8aa96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:59:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56760
I20260929 20:59:35.865484 2416 tablet_service.cc:1514] Processing DeleteTablet for tablet 6965e026f5ef48138c76d87034f8aa96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:59:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33558
I20260929 20:59:35.880865 2734 tablet_replica.cc:330] T 6965e026f5ef48138c76d87034f8aa96 P b44f8fae008f4acf83a4a8b7aef1a2de: stopping tablet replica
I20260929 20:59:35.881125 2734 raft_consensus.cc:2229] T 6965e026f5ef48138c76d87034f8aa96 P b44f8fae008f4acf83a4a8b7aef1a2de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:59:35.881433 2735 tablet_replica.cc:330] T 6965e026f5ef48138c76d87034f8aa96 P 2bc5dec7746f48b99039072a2834da21: stopping tablet replica
I20260929 20:59:35.881405 2734 raft_consensus.cc:2258] T 6965e026f5ef48138c76d87034f8aa96 P b44f8fae008f4acf83a4a8b7aef1a2de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:59:35.881702 2736 tablet_replica.cc:330] T 6965e026f5ef48138c76d87034f8aa96 P 878ec25593c146c9bdc3700d0fc30778: stopping tablet replica
I20260929 20:59:35.881688 2735 raft_consensus.cc:2229] T 6965e026f5ef48138c76d87034f8aa96 P 2bc5dec7746f48b99039072a2834da21 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:59:35.881987 2736 raft_consensus.cc:2229] T 6965e026f5ef48138c76d87034f8aa96 P 878ec25593c146c9bdc3700d0fc30778 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:59:35.882243 2736 raft_consensus.cc:2258] T 6965e026f5ef48138c76d87034f8aa96 P 878ec25593c146c9bdc3700d0fc30778 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:59:35.882486 2735 raft_consensus.cc:2258] T 6965e026f5ef48138c76d87034f8aa96 P 2bc5dec7746f48b99039072a2834da21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:59:35.882543 2734 ts_tablet_manager.cc:1905] T 6965e026f5ef48138c76d87034f8aa96 P b44f8fae008f4acf83a4a8b7aef1a2de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:59:35.883423 2735 ts_tablet_manager.cc:1905] T 6965e026f5ef48138c76d87034f8aa96 P 2bc5dec7746f48b99039072a2834da21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:59:35.883711 2736 ts_tablet_manager.cc:1905] T 6965e026f5ef48138c76d87034f8aa96 P 878ec25593c146c9bdc3700d0fc30778: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:59:35.889197 2734 ts_tablet_manager.cc:1918] T 6965e026f5ef48138c76d87034f8aa96 P b44f8fae008f4acf83a4a8b7aef1a2de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 20:59:35.889325 2734 log.cc:1196] T 6965e026f5ef48138c76d87034f8aa96 P b44f8fae008f4acf83a4a8b7aef1a2de: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-0/wal/wals/6965e026f5ef48138c76d87034f8aa96
I20260929 20:59:35.889746 2734 ts_tablet_manager.cc:1939] T 6965e026f5ef48138c76d87034f8aa96 P b44f8fae008f4acf83a4a8b7aef1a2de: Deleting consensus metadata
I20260929 20:59:35.890882 2735 ts_tablet_manager.cc:1918] T 6965e026f5ef48138c76d87034f8aa96 P 2bc5dec7746f48b99039072a2834da21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 20:59:35.890856 2127 catalog_manager.cc:4841] TS b44f8fae008f4acf83a4a8b7aef1a2de (127.0.97.1:45601): tablet 6965e026f5ef48138c76d87034f8aa96 (table test-table [id=75d51869dbe54bc0a75ee366cde53121]) successfully deleted
I20260929 20:59:35.891100 2735 log.cc:1196] T 6965e026f5ef48138c76d87034f8aa96 P 2bc5dec7746f48b99039072a2834da21: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-2/wal/wals/6965e026f5ef48138c76d87034f8aa96
I20260929 20:59:35.891515 2735 ts_tablet_manager.cc:1939] T 6965e026f5ef48138c76d87034f8aa96 P 2bc5dec7746f48b99039072a2834da21: Deleting consensus metadata
I20260929 20:59:35.892371 2124 catalog_manager.cc:4841] TS 2bc5dec7746f48b99039072a2834da21 (127.0.97.3:46649): tablet 6965e026f5ef48138c76d87034f8aa96 (table test-table [id=75d51869dbe54bc0a75ee366cde53121]) successfully deleted
I20260929 20:59:35.893203 2736 ts_tablet_manager.cc:1918] T 6965e026f5ef48138c76d87034f8aa96 P 878ec25593c146c9bdc3700d0fc30778: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 20:59:35.893353 2736 log.cc:1196] T 6965e026f5ef48138c76d87034f8aa96 P 878ec25593c146c9bdc3700d0fc30778: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-1/wal/wals/6965e026f5ef48138c76d87034f8aa96
I20260929 20:59:35.893836 2736 ts_tablet_manager.cc:1939] T 6965e026f5ef48138c76d87034f8aa96 P 878ec25593c146c9bdc3700d0fc30778: Deleting consensus metadata
I20260929 20:59:35.894858 2127 catalog_manager.cc:4841] TS 878ec25593c146c9bdc3700d0fc30778 (127.0.97.2:33557): tablet 6965e026f5ef48138c76d87034f8aa96 (table test-table [id=75d51869dbe54bc0a75ee366cde53121]) successfully deleted
Sep 29 20:59:35 dist-test-slave-1dms krb5kdc[2095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715575, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 20:59:35 dist-test-slave-1dms krb5kdc[2095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715575, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:59:35.933248 2744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:35.910219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52736 (local address 127.0.97.62:40803)
0929 20:59:35.911091 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:35.911097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:35.911127 (+ 30us) server_negotiation.cc:407] Connection header received
0929 20:59:35.911180 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:35.911184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:35.911241 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:35.911340 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:35.913282 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:35.914040 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:35.915424 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:35.915813 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:35.915914 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:35.927814 (+ 11900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:35.927820 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:35.927823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:35.927872 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:35.930594 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:35.931335 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:35.931340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:35.931343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:35.931495 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:35.931877 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:35.931881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:35.931884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:35.932219 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:35.932419 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:35.932674 (+ 255us) server_negotiation.cc:299] Negotiation successful
0929 20:59:35.932945 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":738,"thread_start_us":594,"threads_started":1}
I20260929 20:59:35.936502 2140 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:59:35.937100 2140 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:59:35.942867 2276 tablet_service.cc:1467] Processing CreateTablet for tablet 610d91ffda634798bb008ab184d247e7 (DEFAULT_TABLE table=test-table [id=07b3327c54064181bd581a65918835bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:59:35.942941 2416 tablet_service.cc:1467] Processing CreateTablet for tablet 610d91ffda634798bb008ab184d247e7 (DEFAULT_TABLE table=test-table [id=07b3327c54064181bd581a65918835bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:59:35.943032 2555 tablet_service.cc:1467] Processing CreateTablet for tablet 610d91ffda634798bb008ab184d247e7 (DEFAULT_TABLE table=test-table [id=07b3327c54064181bd581a65918835bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:59:35.943328 2276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 610d91ffda634798bb008ab184d247e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:35.943387 2416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 610d91ffda634798bb008ab184d247e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:35.943391 2555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 610d91ffda634798bb008ab184d247e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:35.949596 2745 tablet_bootstrap.cc:492] T 610d91ffda634798bb008ab184d247e7 P b44f8fae008f4acf83a4a8b7aef1a2de: Bootstrap starting.
I20260929 20:59:35.950431 2747 tablet_bootstrap.cc:492] T 610d91ffda634798bb008ab184d247e7 P 878ec25593c146c9bdc3700d0fc30778: Bootstrap starting.
I20260929 20:59:35.951210 2746 tablet_bootstrap.cc:492] T 610d91ffda634798bb008ab184d247e7 P 2bc5dec7746f48b99039072a2834da21: Bootstrap starting.
I20260929 20:59:35.951524 2745 tablet_bootstrap.cc:654] T 610d91ffda634798bb008ab184d247e7 P b44f8fae008f4acf83a4a8b7aef1a2de: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:35.952057 2747 tablet_bootstrap.cc:654] T 610d91ffda634798bb008ab184d247e7 P 878ec25593c146c9bdc3700d0fc30778: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:35.952769 2746 tablet_bootstrap.cc:654] T 610d91ffda634798bb008ab184d247e7 P 2bc5dec7746f48b99039072a2834da21: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:35.954584 2745 tablet_bootstrap.cc:492] T 610d91ffda634798bb008ab184d247e7 P b44f8fae008f4acf83a4a8b7aef1a2de: No bootstrap required, opened a new log
I20260929 20:59:35.954727 2745 ts_tablet_manager.cc:1397] T 610d91ffda634798bb008ab184d247e7 P b44f8fae008f4acf83a4a8b7aef1a2de: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 20:59:35.955060 2746 tablet_bootstrap.cc:492] T 610d91ffda634798bb008ab184d247e7 P 2bc5dec7746f48b99039072a2834da21: No bootstrap required, opened a new log
I20260929 20:59:35.955087 2747 tablet_bootstrap.cc:492] T 610d91ffda634798bb008ab184d247e7 P 878ec25593c146c9bdc3700d0fc30778: No bootstrap required, opened a new log
I20260929 20:59:35.955202 2747 ts_tablet_manager.cc:1397] T 610d91ffda634798bb008ab184d247e7 P 878ec25593c146c9bdc3700d0fc30778: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260929 20:59:35.955204 2746 ts_tablet_manager.cc:1397] T 610d91ffda634798bb008ab184d247e7 P 2bc5dec7746f48b99039072a2834da21: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260929 20:59:35.955476 2745 raft_consensus.cc:348] T 610d91ffda634798bb008ab184d247e7 P b44f8fae008f4acf83a4a8b7aef1a2de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } }
I20260929 20:59:35.955722 2745 raft_consensus.cc:374] T 610d91ffda634798bb008ab184d247e7 P b44f8fae008f4acf83a4a8b7aef1a2de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:35.955825 2745 raft_consensus.cc:729] T 610d91ffda634798bb008ab184d247e7 P b44f8fae008f4acf83a4a8b7aef1a2de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b44f8fae008f4acf83a4a8b7aef1a2de, State: Initialized, Role: FOLLOWER
I20260929 20:59:35.955847 2746 raft_consensus.cc:348] T 610d91ffda634798bb008ab184d247e7 P 2bc5dec7746f48b99039072a2834da21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } }
I20260929 20:59:35.955932 2747 raft_consensus.cc:348] T 610d91ffda634798bb008ab184d247e7 P 878ec25593c146c9bdc3700d0fc30778 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } }
I20260929 20:59:35.956055 2746 raft_consensus.cc:374] T 610d91ffda634798bb008ab184d247e7 P 2bc5dec7746f48b99039072a2834da21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:35.956128 2747 raft_consensus.cc:374] T 610d91ffda634798bb008ab184d247e7 P 878ec25593c146c9bdc3700d0fc30778 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:35.956149 2746 raft_consensus.cc:729] T 610d91ffda634798bb008ab184d247e7 P 2bc5dec7746f48b99039072a2834da21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bc5dec7746f48b99039072a2834da21, State: Initialized, Role: FOLLOWER
I20260929 20:59:35.956041 2745 consensus_queue.cc:260] T 610d91ffda634798bb008ab184d247e7 P b44f8fae008f4acf83a4a8b7aef1a2de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } }
I20260929 20:59:35.956203 2747 raft_consensus.cc:729] T 610d91ffda634798bb008ab184d247e7 P 878ec25593c146c9bdc3700d0fc30778 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 878ec25593c146c9bdc3700d0fc30778, State: Initialized, Role: FOLLOWER
I20260929 20:59:35.956391 2746 consensus_queue.cc:260] T 610d91ffda634798bb008ab184d247e7 P 2bc5dec7746f48b99039072a2834da21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } }
I20260929 20:59:35.956429 2747 consensus_queue.cc:260] T 610d91ffda634798bb008ab184d247e7 P 878ec25593c146c9bdc3700d0fc30778 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } }
I20260929 20:59:35.957506 2746 ts_tablet_manager.cc:1428] T 610d91ffda634798bb008ab184d247e7 P 2bc5dec7746f48b99039072a2834da21: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 20:59:35.958118 2745 ts_tablet_manager.cc:1428] T 610d91ffda634798bb008ab184d247e7 P b44f8fae008f4acf83a4a8b7aef1a2de: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260929 20:59:35.958439 2747 ts_tablet_manager.cc:1428] T 610d91ffda634798bb008ab184d247e7 P 878ec25593c146c9bdc3700d0fc30778: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260929 20:59:36.039776 2681 raft_consensus.cc:482] T 610d91ffda634798bb008ab184d247e7 P 878ec25593c146c9bdc3700d0fc30778 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:59:36.039980 2681 raft_consensus.cc:504] T 610d91ffda634798bb008ab184d247e7 P 878ec25593c146c9bdc3700d0fc30778 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } }
I20260929 20:59:36.040671 2681 leader_election.cc:288] T 610d91ffda634798bb008ab184d247e7 P 878ec25593c146c9bdc3700d0fc30778 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b44f8fae008f4acf83a4a8b7aef1a2de (127.0.97.1:45601), 2bc5dec7746f48b99039072a2834da21 (127.0.97.3:46649)
I20260929 20:59:36.041249 2302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "610d91ffda634798bb008ab184d247e7" candidate_uuid: "878ec25593c146c9bdc3700d0fc30778" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" is_pre_election: true
I20260929 20:59:36.041519 2302 raft_consensus.cc:2454] T 610d91ffda634798bb008ab184d247e7 P b44f8fae008f4acf83a4a8b7aef1a2de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 878ec25593c146c9bdc3700d0fc30778 in term 0.
I20260929 20:59:36.041675 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "610d91ffda634798bb008ab184d247e7" candidate_uuid: "878ec25593c146c9bdc3700d0fc30778" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bc5dec7746f48b99039072a2834da21" is_pre_election: true
I20260929 20:59:36.041961 2574 raft_consensus.cc:2454] T 610d91ffda634798bb008ab184d247e7 P 2bc5dec7746f48b99039072a2834da21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 878ec25593c146c9bdc3700d0fc30778 in term 0.
I20260929 20:59:36.041996 2370 leader_election.cc:302] T 610d91ffda634798bb008ab184d247e7 P 878ec25593c146c9bdc3700d0fc30778 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 878ec25593c146c9bdc3700d0fc30778, b44f8fae008f4acf83a4a8b7aef1a2de; no voters:
I20260929 20:59:36.042335 2681 raft_consensus.cc:2781] T 610d91ffda634798bb008ab184d247e7 P 878ec25593c146c9bdc3700d0fc30778 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:59:36.042469 2681 raft_consensus.cc:482] T 610d91ffda634798bb008ab184d247e7 P 878ec25593c146c9bdc3700d0fc30778 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:59:36.042580 2681 raft_consensus.cc:3037] T 610d91ffda634798bb008ab184d247e7 P 878ec25593c146c9bdc3700d0fc30778 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:36.044497 2681 raft_consensus.cc:504] T 610d91ffda634798bb008ab184d247e7 P 878ec25593c146c9bdc3700d0fc30778 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } }
I20260929 20:59:36.045046 2681 leader_election.cc:288] T 610d91ffda634798bb008ab184d247e7 P 878ec25593c146c9bdc3700d0fc30778 [CANDIDATE]: Term 1 election: Requested vote from peers b44f8fae008f4acf83a4a8b7aef1a2de (127.0.97.1:45601), 2bc5dec7746f48b99039072a2834da21 (127.0.97.3:46649)
I20260929 20:59:36.045413 2302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "610d91ffda634798bb008ab184d247e7" candidate_uuid: "878ec25593c146c9bdc3700d0fc30778" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de"
I20260929 20:59:36.045482 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "610d91ffda634798bb008ab184d247e7" candidate_uuid: "878ec25593c146c9bdc3700d0fc30778" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bc5dec7746f48b99039072a2834da21"
I20260929 20:59:36.045595 2302 raft_consensus.cc:3037] T 610d91ffda634798bb008ab184d247e7 P b44f8fae008f4acf83a4a8b7aef1a2de [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:36.045704 2574 raft_consensus.cc:3037] T 610d91ffda634798bb008ab184d247e7 P 2bc5dec7746f48b99039072a2834da21 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:36.047499 2302 raft_consensus.cc:2454] T 610d91ffda634798bb008ab184d247e7 P b44f8fae008f4acf83a4a8b7aef1a2de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 878ec25593c146c9bdc3700d0fc30778 in term 1.
I20260929 20:59:36.047572 2574 raft_consensus.cc:2454] T 610d91ffda634798bb008ab184d247e7 P 2bc5dec7746f48b99039072a2834da21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 878ec25593c146c9bdc3700d0fc30778 in term 1.
I20260929 20:59:36.047868 2370 leader_election.cc:302] T 610d91ffda634798bb008ab184d247e7 P 878ec25593c146c9bdc3700d0fc30778 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 878ec25593c146c9bdc3700d0fc30778, b44f8fae008f4acf83a4a8b7aef1a2de; no voters:
I20260929 20:59:36.048148 2681 raft_consensus.cc:2781] T 610d91ffda634798bb008ab184d247e7 P 878ec25593c146c9bdc3700d0fc30778 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:59:36.048287 2681 raft_consensus.cc:686] T 610d91ffda634798bb008ab184d247e7 P 878ec25593c146c9bdc3700d0fc30778 [term 1 LEADER]: Becoming Leader. State: Replica: 878ec25593c146c9bdc3700d0fc30778, State: Running, Role: LEADER
I20260929 20:59:36.048527 2681 consensus_queue.cc:237] T 610d91ffda634798bb008ab184d247e7 P 878ec25593c146c9bdc3700d0fc30778 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } }
I20260929 20:59:36.051059 2140 catalog_manager.cc:5495] T 610d91ffda634798bb008ab184d247e7 P 878ec25593c146c9bdc3700d0fc30778 reported cstate change: term changed from 0 to 1, leader changed from <none> to 878ec25593c146c9bdc3700d0fc30778 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "878ec25593c146c9bdc3700d0fc30778" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } health_report { overall_health: UNKNOWN } } }
I20260929 20:59:36.083171 2754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:36.078218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33570 (local address 127.0.97.2:33557)
0929 20:59:36.078734 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:36.078738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:36.078758 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:59:36.078807 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:36.078811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:36.078864 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:36.078967 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:59:36.079336 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:36.080129 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:36.081623 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:36.082016 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:36.082144 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:36.082538 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:59:36.082681 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:59:36.082759 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:36.082877 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 20:59:36.082935 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":254,"threads_started":1}
I20260929 20:59:36.095940 2754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:36.091024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36487 (local address 127.0.97.2:33557)
0929 20:59:36.091220 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:36.091226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:36.091407 (+ 181us) server_negotiation.cc:407] Connection header received
0929 20:59:36.091576 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:36.091581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:36.091635 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:36.091718 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:59:36.092097 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:36.092752 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:36.094424 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:36.094798 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:36.094927 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:36.095241 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:59:36.095328 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:59:36.095410 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:36.095512 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 20:59:36.095631 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":75}
I20260929 20:59:36.096010 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:36.090807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33557 (local address 127.0.97.2:36487)
0929 20:59:36.091322 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:36.091337 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:36.091473 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:36.091726 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:36.091730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:36.091740 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:59:36.091969 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:36.091977 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:36.092902 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:36.092909 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:36.094297 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:36.094308 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:36.094942 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:36.094947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:36.095077 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:36.095108 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:59:36.095420 (+ 312us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:59:36.095430 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 20:59:36.095500 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 20:59:36.095681 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":44,"thread_start_us":249,"threads_started":1}
I20260929 20:59:36.104331 2302 raft_consensus.cc:1261] T 610d91ffda634798bb008ab184d247e7 P b44f8fae008f4acf83a4a8b7aef1a2de [term 1 FOLLOWER]: Refusing update from remote peer 878ec25593c146c9bdc3700d0fc30778: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:59:36.104357 2574 raft_consensus.cc:1261] T 610d91ffda634798bb008ab184d247e7 P 2bc5dec7746f48b99039072a2834da21 [term 1 FOLLOWER]: Refusing update from remote peer 878ec25593c146c9bdc3700d0fc30778: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:59:36.104863 2715 consensus_queue.cc:1035] T 610d91ffda634798bb008ab184d247e7 P 878ec25593c146c9bdc3700d0fc30778 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:59:36.105265 2720 consensus_queue.cc:1035] T 610d91ffda634798bb008ab184d247e7 P 878ec25593c146c9bdc3700d0fc30778 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:59:36.151969 2138 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:59:36.152218 2138 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:59:36.155350 2138 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d36341f2e2ba4bccbbae2e9c386e6696: Sending DeleteTablet for 3 replicas of tablet 610d91ffda634798bb008ab184d247e7
I20260929 20:59:36.156208 2555 tablet_service.cc:1514] Processing DeleteTablet for tablet 610d91ffda634798bb008ab184d247e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:59:36 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56760
I20260929 20:59:36.156244 2415 tablet_service.cc:1514] Processing DeleteTablet for tablet 610d91ffda634798bb008ab184d247e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:59:36 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33558
I20260929 20:59:36.156484 2735 tablet_replica.cc:330] T 610d91ffda634798bb008ab184d247e7 P 2bc5dec7746f48b99039072a2834da21: stopping tablet replica
I20260929 20:59:36.156704 2735 raft_consensus.cc:2229] T 610d91ffda634798bb008ab184d247e7 P 2bc5dec7746f48b99039072a2834da21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:59:36.156996 2736 tablet_replica.cc:330] T 610d91ffda634798bb008ab184d247e7 P 878ec25593c146c9bdc3700d0fc30778: stopping tablet replica
I20260929 20:59:36.157219 2736 raft_consensus.cc:2229] T 610d91ffda634798bb008ab184d247e7 P 878ec25593c146c9bdc3700d0fc30778 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:59:36.157061 2138 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:59:36.157719 2138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:59:36.157737 2736 raft_consensus.cc:2258] T 610d91ffda634798bb008ab184d247e7 P 878ec25593c146c9bdc3700d0fc30778 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:59:36.158301 2735 raft_consensus.cc:2258] T 610d91ffda634798bb008ab184d247e7 P 2bc5dec7746f48b99039072a2834da21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:59:36.158306 2276 tablet_service.cc:1514] Processing DeleteTablet for tablet 610d91ffda634798bb008ab184d247e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:59:36 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39190
I20260929 20:59:36.158557 2734 tablet_replica.cc:330] T 610d91ffda634798bb008ab184d247e7 P b44f8fae008f4acf83a4a8b7aef1a2de: stopping tablet replica
I20260929 20:59:36.158749 2734 raft_consensus.cc:2229] T 610d91ffda634798bb008ab184d247e7 P b44f8fae008f4acf83a4a8b7aef1a2de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:59:36.158947 2734 raft_consensus.cc:2258] T 610d91ffda634798bb008ab184d247e7 P b44f8fae008f4acf83a4a8b7aef1a2de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:59:36.159473 2735 ts_tablet_manager.cc:1905] T 610d91ffda634798bb008ab184d247e7 P 2bc5dec7746f48b99039072a2834da21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:59:36.161459 2736 ts_tablet_manager.cc:1905] T 610d91ffda634798bb008ab184d247e7 P 878ec25593c146c9bdc3700d0fc30778: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:59:36.165436 2555 tablet_service.cc:1467] Processing CreateTablet for tablet 69d3bf1fecd54e64988f167c370d061c (DEFAULT_TABLE table=test-table [id=6d2c564ff1cd47bdbb0f45ddc779b1a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:59:36.165467 2276 tablet_service.cc:1467] Processing CreateTablet for tablet 69d3bf1fecd54e64988f167c370d061c (DEFAULT_TABLE table=test-table [id=6d2c564ff1cd47bdbb0f45ddc779b1a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:59:36.165767 2415 tablet_service.cc:1467] Processing CreateTablet for tablet 69d3bf1fecd54e64988f167c370d061c (DEFAULT_TABLE table=test-table [id=6d2c564ff1cd47bdbb0f45ddc779b1a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:59:36.165884 2276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69d3bf1fecd54e64988f167c370d061c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:36.165900 2555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69d3bf1fecd54e64988f167c370d061c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:36.166920 2734 ts_tablet_manager.cc:1905] T 610d91ffda634798bb008ab184d247e7 P b44f8fae008f4acf83a4a8b7aef1a2de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:59:36.172439 2745 tablet_bootstrap.cc:492] T 69d3bf1fecd54e64988f167c370d061c P b44f8fae008f4acf83a4a8b7aef1a2de: Bootstrap starting.
I20260929 20:59:36.173308 2734 ts_tablet_manager.cc:1918] T 610d91ffda634798bb008ab184d247e7 P b44f8fae008f4acf83a4a8b7aef1a2de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:59:36.173475 2734 log.cc:1196] T 610d91ffda634798bb008ab184d247e7 P b44f8fae008f4acf83a4a8b7aef1a2de: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-0/wal/wals/610d91ffda634798bb008ab184d247e7
I20260929 20:59:36.173959 2734 ts_tablet_manager.cc:1939] T 610d91ffda634798bb008ab184d247e7 P b44f8fae008f4acf83a4a8b7aef1a2de: Deleting consensus metadata
I20260929 20:59:36.174178 2746 tablet_bootstrap.cc:492] T 69d3bf1fecd54e64988f167c370d061c P 2bc5dec7746f48b99039072a2834da21: Bootstrap starting.
I20260929 20:59:36.174429 2415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69d3bf1fecd54e64988f167c370d061c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:36.174624 2745 tablet_bootstrap.cc:654] T 69d3bf1fecd54e64988f167c370d061c P b44f8fae008f4acf83a4a8b7aef1a2de: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:36.175673 2746 tablet_bootstrap.cc:654] T 69d3bf1fecd54e64988f167c370d061c P 2bc5dec7746f48b99039072a2834da21: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:36.176594 2745 tablet_bootstrap.cc:492] T 69d3bf1fecd54e64988f167c370d061c P b44f8fae008f4acf83a4a8b7aef1a2de: No bootstrap required, opened a new log
I20260929 20:59:36.176739 2745 ts_tablet_manager.cc:1397] T 69d3bf1fecd54e64988f167c370d061c P b44f8fae008f4acf83a4a8b7aef1a2de: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260929 20:59:36.176550 2127 catalog_manager.cc:4841] TS b44f8fae008f4acf83a4a8b7aef1a2de (127.0.97.1:45601): tablet 610d91ffda634798bb008ab184d247e7 (table test-table [id=07b3327c54064181bd581a65918835bd]) successfully deleted
I20260929 20:59:36.177107 2735 ts_tablet_manager.cc:1918] T 610d91ffda634798bb008ab184d247e7 P 2bc5dec7746f48b99039072a2834da21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:59:36.177261 2735 log.cc:1196] T 610d91ffda634798bb008ab184d247e7 P 2bc5dec7746f48b99039072a2834da21: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-2/wal/wals/610d91ffda634798bb008ab184d247e7
I20260929 20:59:36.177645 2735 ts_tablet_manager.cc:1939] T 610d91ffda634798bb008ab184d247e7 P 2bc5dec7746f48b99039072a2834da21: Deleting consensus metadata
I20260929 20:59:36.177910 2745 raft_consensus.cc:348] T 69d3bf1fecd54e64988f167c370d061c P b44f8fae008f4acf83a4a8b7aef1a2de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } }
I20260929 20:59:36.178144 2745 raft_consensus.cc:374] T 69d3bf1fecd54e64988f167c370d061c P b44f8fae008f4acf83a4a8b7aef1a2de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:36.178223 2745 raft_consensus.cc:729] T 69d3bf1fecd54e64988f167c370d061c P b44f8fae008f4acf83a4a8b7aef1a2de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b44f8fae008f4acf83a4a8b7aef1a2de, State: Initialized, Role: FOLLOWER
I20260929 20:59:36.178463 2745 consensus_queue.cc:260] T 69d3bf1fecd54e64988f167c370d061c P b44f8fae008f4acf83a4a8b7aef1a2de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } }
I20260929 20:59:36.178726 2124 catalog_manager.cc:4841] TS 2bc5dec7746f48b99039072a2834da21 (127.0.97.3:46649): tablet 610d91ffda634798bb008ab184d247e7 (table test-table [id=07b3327c54064181bd581a65918835bd]) successfully deleted
I20260929 20:59:36.178923 2745 ts_tablet_manager.cc:1428] T 69d3bf1fecd54e64988f167c370d061c P b44f8fae008f4acf83a4a8b7aef1a2de: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 20:59:36.179975 2746 tablet_bootstrap.cc:492] T 69d3bf1fecd54e64988f167c370d061c P 2bc5dec7746f48b99039072a2834da21: No bootstrap required, opened a new log
I20260929 20:59:36.180126 2746 ts_tablet_manager.cc:1397] T 69d3bf1fecd54e64988f167c370d061c P 2bc5dec7746f48b99039072a2834da21: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20260929 20:59:36.180930 2746 raft_consensus.cc:348] T 69d3bf1fecd54e64988f167c370d061c P 2bc5dec7746f48b99039072a2834da21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } }
I20260929 20:59:36.181167 2746 raft_consensus.cc:374] T 69d3bf1fecd54e64988f167c370d061c P 2bc5dec7746f48b99039072a2834da21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:36.181311 2746 raft_consensus.cc:729] T 69d3bf1fecd54e64988f167c370d061c P 2bc5dec7746f48b99039072a2834da21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bc5dec7746f48b99039072a2834da21, State: Initialized, Role: FOLLOWER
I20260929 20:59:36.181608 2746 consensus_queue.cc:260] T 69d3bf1fecd54e64988f167c370d061c P 2bc5dec7746f48b99039072a2834da21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } }
I20260929 20:59:36.182971 2736 ts_tablet_manager.cc:1918] T 610d91ffda634798bb008ab184d247e7 P 878ec25593c146c9bdc3700d0fc30778: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:59:36.183128 2736 log.cc:1196] T 610d91ffda634798bb008ab184d247e7 P 878ec25593c146c9bdc3700d0fc30778: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-1/wal/wals/610d91ffda634798bb008ab184d247e7
I20260929 20:59:36.183216 2746 ts_tablet_manager.cc:1428] T 69d3bf1fecd54e64988f167c370d061c P 2bc5dec7746f48b99039072a2834da21: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 20:59:36.183521 2736 ts_tablet_manager.cc:1939] T 610d91ffda634798bb008ab184d247e7 P 878ec25593c146c9bdc3700d0fc30778: Deleting consensus metadata
I20260929 20:59:36.184340 2747 tablet_bootstrap.cc:492] T 69d3bf1fecd54e64988f167c370d061c P 878ec25593c146c9bdc3700d0fc30778: Bootstrap starting.
I20260929 20:59:36.186084 2747 tablet_bootstrap.cc:654] T 69d3bf1fecd54e64988f167c370d061c P 878ec25593c146c9bdc3700d0fc30778: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:36.187134 2127 catalog_manager.cc:4841] TS 878ec25593c146c9bdc3700d0fc30778 (127.0.97.2:33557): tablet 610d91ffda634798bb008ab184d247e7 (table test-table [id=07b3327c54064181bd581a65918835bd]) successfully deleted
I20260929 20:59:36.187901 2747 tablet_bootstrap.cc:492] T 69d3bf1fecd54e64988f167c370d061c P 878ec25593c146c9bdc3700d0fc30778: No bootstrap required, opened a new log
I20260929 20:59:36.188063 2747 ts_tablet_manager.cc:1397] T 69d3bf1fecd54e64988f167c370d061c P 878ec25593c146c9bdc3700d0fc30778: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260929 20:59:36.188632 2747 raft_consensus.cc:348] T 69d3bf1fecd54e64988f167c370d061c P 878ec25593c146c9bdc3700d0fc30778 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } }
I20260929 20:59:36.188906 2747 raft_consensus.cc:374] T 69d3bf1fecd54e64988f167c370d061c P 878ec25593c146c9bdc3700d0fc30778 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:36.189007 2747 raft_consensus.cc:729] T 69d3bf1fecd54e64988f167c370d061c P 878ec25593c146c9bdc3700d0fc30778 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 878ec25593c146c9bdc3700d0fc30778, State: Initialized, Role: FOLLOWER
I20260929 20:59:36.189242 2747 consensus_queue.cc:260] T 69d3bf1fecd54e64988f167c370d061c P 878ec25593c146c9bdc3700d0fc30778 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } }
I20260929 20:59:36.189679 2747 ts_tablet_manager.cc:1428] T 69d3bf1fecd54e64988f167c370d061c P 878ec25593c146c9bdc3700d0fc30778: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260929 20:59:36.255036 2651 raft_consensus.cc:482] T 69d3bf1fecd54e64988f167c370d061c P 2bc5dec7746f48b99039072a2834da21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:59:36.255264 2651 raft_consensus.cc:504] T 69d3bf1fecd54e64988f167c370d061c P 2bc5dec7746f48b99039072a2834da21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } }
I20260929 20:59:36.255856 2651 leader_election.cc:288] T 69d3bf1fecd54e64988f167c370d061c P 2bc5dec7746f48b99039072a2834da21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b44f8fae008f4acf83a4a8b7aef1a2de (127.0.97.1:45601), 878ec25593c146c9bdc3700d0fc30778 (127.0.97.2:33557)
I20260929 20:59:36.256448 2441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d3bf1fecd54e64988f167c370d061c" candidate_uuid: "2bc5dec7746f48b99039072a2834da21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "878ec25593c146c9bdc3700d0fc30778" is_pre_election: true
I20260929 20:59:36.256467 2302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d3bf1fecd54e64988f167c370d061c" candidate_uuid: "2bc5dec7746f48b99039072a2834da21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" is_pre_election: true
I20260929 20:59:36.256695 2441 raft_consensus.cc:2454] T 69d3bf1fecd54e64988f167c370d061c P 878ec25593c146c9bdc3700d0fc30778 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bc5dec7746f48b99039072a2834da21 in term 0.
I20260929 20:59:36.256726 2302 raft_consensus.cc:2454] T 69d3bf1fecd54e64988f167c370d061c P b44f8fae008f4acf83a4a8b7aef1a2de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bc5dec7746f48b99039072a2834da21 in term 0.
I20260929 20:59:36.257153 2509 leader_election.cc:302] T 69d3bf1fecd54e64988f167c370d061c P 2bc5dec7746f48b99039072a2834da21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bc5dec7746f48b99039072a2834da21, b44f8fae008f4acf83a4a8b7aef1a2de; no voters:
I20260929 20:59:36.257429 2651 raft_consensus.cc:2781] T 69d3bf1fecd54e64988f167c370d061c P 2bc5dec7746f48b99039072a2834da21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:59:36.257546 2651 raft_consensus.cc:482] T 69d3bf1fecd54e64988f167c370d061c P 2bc5dec7746f48b99039072a2834da21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:59:36.257658 2651 raft_consensus.cc:3037] T 69d3bf1fecd54e64988f167c370d061c P 2bc5dec7746f48b99039072a2834da21 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:36.260061 2651 raft_consensus.cc:504] T 69d3bf1fecd54e64988f167c370d061c P 2bc5dec7746f48b99039072a2834da21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } }
I20260929 20:59:36.260564 2651 leader_election.cc:288] T 69d3bf1fecd54e64988f167c370d061c P 2bc5dec7746f48b99039072a2834da21 [CANDIDATE]: Term 1 election: Requested vote from peers b44f8fae008f4acf83a4a8b7aef1a2de (127.0.97.1:45601), 878ec25593c146c9bdc3700d0fc30778 (127.0.97.2:33557)
I20260929 20:59:36.261066 2441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d3bf1fecd54e64988f167c370d061c" candidate_uuid: "2bc5dec7746f48b99039072a2834da21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "878ec25593c146c9bdc3700d0fc30778"
I20260929 20:59:36.261251 2441 raft_consensus.cc:3037] T 69d3bf1fecd54e64988f167c370d061c P 878ec25593c146c9bdc3700d0fc30778 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:36.263377 2441 raft_consensus.cc:2454] T 69d3bf1fecd54e64988f167c370d061c P 878ec25593c146c9bdc3700d0fc30778 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bc5dec7746f48b99039072a2834da21 in term 1.
I20260929 20:59:36.263731 2509 leader_election.cc:302] T 69d3bf1fecd54e64988f167c370d061c P 2bc5dec7746f48b99039072a2834da21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bc5dec7746f48b99039072a2834da21, 878ec25593c146c9bdc3700d0fc30778; no voters:
I20260929 20:59:36.263716 2302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d3bf1fecd54e64988f167c370d061c" candidate_uuid: "2bc5dec7746f48b99039072a2834da21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de"
I20260929 20:59:36.263931 2302 raft_consensus.cc:3037] T 69d3bf1fecd54e64988f167c370d061c P b44f8fae008f4acf83a4a8b7aef1a2de [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:36.263984 2651 raft_consensus.cc:2781] T 69d3bf1fecd54e64988f167c370d061c P 2bc5dec7746f48b99039072a2834da21 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:59:36.264148 2651 raft_consensus.cc:686] T 69d3bf1fecd54e64988f167c370d061c P 2bc5dec7746f48b99039072a2834da21 [term 1 LEADER]: Becoming Leader. State: Replica: 2bc5dec7746f48b99039072a2834da21, State: Running, Role: LEADER
I20260929 20:59:36.264394 2651 consensus_queue.cc:237] T 69d3bf1fecd54e64988f167c370d061c P 2bc5dec7746f48b99039072a2834da21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } }
I20260929 20:59:36.267105 2138 catalog_manager.cc:5495] T 69d3bf1fecd54e64988f167c370d061c P 2bc5dec7746f48b99039072a2834da21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2bc5dec7746f48b99039072a2834da21 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "2bc5dec7746f48b99039072a2834da21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bc5dec7746f48b99039072a2834da21" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46649 } health_report { overall_health: HEALTHY } } }
I20260929 20:59:36.267772 2302 raft_consensus.cc:2454] T 69d3bf1fecd54e64988f167c370d061c P b44f8fae008f4acf83a4a8b7aef1a2de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bc5dec7746f48b99039072a2834da21 in term 1.
I20260929 20:59:36.292240 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:36.282738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56776 (local address 127.0.97.3:46649)
0929 20:59:36.286715 (+ 3977us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:36.286722 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:36.286747 (+ 25us) server_negotiation.cc:407] Connection header received
0929 20:59:36.286815 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:36.286821 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:36.286902 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:36.287011 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:59:36.287433 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:36.288587 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:36.289888 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:36.290538 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:36.290661 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:36.291471 (+ 810us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:59:36.291597 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:59:36.291682 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:36.291811 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 20:59:36.291884 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3826,"thread_start_us":3670,"threads_started":1}
I20260929 20:59:36.302438 2441 raft_consensus.cc:1261] T 69d3bf1fecd54e64988f167c370d061c P 878ec25593c146c9bdc3700d0fc30778 [term 1 FOLLOWER]: Refusing update from remote peer 2bc5dec7746f48b99039072a2834da21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:59:36.302881 2302 raft_consensus.cc:1261] T 69d3bf1fecd54e64988f167c370d061c P b44f8fae008f4acf83a4a8b7aef1a2de [term 1 FOLLOWER]: Refusing update from remote peer 2bc5dec7746f48b99039072a2834da21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:59:36.302976 2725 consensus_queue.cc:1035] T 69d3bf1fecd54e64988f167c370d061c P 2bc5dec7746f48b99039072a2834da21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "878ec25593c146c9bdc3700d0fc30778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:59:36.303422 2651 consensus_queue.cc:1035] T 69d3bf1fecd54e64988f167c370d061c P 2bc5dec7746f48b99039072a2834da21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b44f8fae008f4acf83a4a8b7aef1a2de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 20:59:36.324380 2716 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 20:59:37.903887 2138 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:59:37.904085 2138 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:59:37.907706 2138 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d36341f2e2ba4bccbbae2e9c386e6696: Sending DeleteTablet for 3 replicas of tablet 69d3bf1fecd54e64988f167c370d061c
I20260929 20:59:37.908422 2276 tablet_service.cc:1514] Processing DeleteTablet for tablet 69d3bf1fecd54e64988f167c370d061c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:59:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39190
I20260929 20:59:37.908465 2415 tablet_service.cc:1514] Processing DeleteTablet for tablet 69d3bf1fecd54e64988f167c370d061c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:59:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33558
I20260929 20:59:37.908854 2555 tablet_service.cc:1514] Processing DeleteTablet for tablet 69d3bf1fecd54e64988f167c370d061c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:59:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56760
I20260929 20:59:37.909053 2792 tablet_replica.cc:330] T 69d3bf1fecd54e64988f167c370d061c P 878ec25593c146c9bdc3700d0fc30778: stopping tablet replica
I20260929 20:59:37.909215 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 2211
I20260929 20:59:37.909238 2793 tablet_replica.cc:330] T 69d3bf1fecd54e64988f167c370d061c P b44f8fae008f4acf83a4a8b7aef1a2de: stopping tablet replica
I20260929 20:59:37.909242 2792 raft_consensus.cc:2229] T 69d3bf1fecd54e64988f167c370d061c P 878ec25593c146c9bdc3700d0fc30778 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:59:37.909457 2792 raft_consensus.cc:2258] T 69d3bf1fecd54e64988f167c370d061c P 878ec25593c146c9bdc3700d0fc30778 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:59:37.909473 2793 raft_consensus.cc:2229] T 69d3bf1fecd54e64988f167c370d061c P b44f8fae008f4acf83a4a8b7aef1a2de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:59:37.909760 2793 raft_consensus.cc:2258] T 69d3bf1fecd54e64988f167c370d061c P b44f8fae008f4acf83a4a8b7aef1a2de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:59:37.910445 2792 ts_tablet_manager.cc:1905] T 69d3bf1fecd54e64988f167c370d061c P 878ec25593c146c9bdc3700d0fc30778: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:59:37.917026 2793 ts_tablet_manager.cc:1905] T 69d3bf1fecd54e64988f167c370d061c P b44f8fae008f4acf83a4a8b7aef1a2de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:59:37.918519 2792 ts_tablet_manager.cc:1918] T 69d3bf1fecd54e64988f167c370d061c P 878ec25593c146c9bdc3700d0fc30778: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 20:59:37.918748 2792 log.cc:1196] T 69d3bf1fecd54e64988f167c370d061c P 878ec25593c146c9bdc3700d0fc30778: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-1/wal/wals/69d3bf1fecd54e64988f167c370d061c
I20260929 20:59:37.919853 2792 ts_tablet_manager.cc:1939] T 69d3bf1fecd54e64988f167c370d061c P 878ec25593c146c9bdc3700d0fc30778: Deleting consensus metadata
I20260929 20:59:37.920567 2794 tablet_replica.cc:330] T 69d3bf1fecd54e64988f167c370d061c P 2bc5dec7746f48b99039072a2834da21: stopping tablet replica
I20260929 20:59:37.920861 2794 raft_consensus.cc:2229] T 69d3bf1fecd54e64988f167c370d061c P 2bc5dec7746f48b99039072a2834da21 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:59:37.921237 2127 catalog_manager.cc:4841] TS 878ec25593c146c9bdc3700d0fc30778 (127.0.97.2:33557): tablet 69d3bf1fecd54e64988f167c370d061c (table test-table [id=6d2c564ff1cd47bdbb0f45ddc779b1a3]) successfully deleted
I20260929 20:59:37.921316 2794 raft_consensus.cc:2258] T 69d3bf1fecd54e64988f167c370d061c P 2bc5dec7746f48b99039072a2834da21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:59:37.923249 2794 ts_tablet_manager.cc:1905] T 69d3bf1fecd54e64988f167c370d061c P 2bc5dec7746f48b99039072a2834da21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:59:37.927138 2793 ts_tablet_manager.cc:1918] T 69d3bf1fecd54e64988f167c370d061c P b44f8fae008f4acf83a4a8b7aef1a2de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 20:59:37.927301 2793 log.cc:1196] T 69d3bf1fecd54e64988f167c370d061c P b44f8fae008f4acf83a4a8b7aef1a2de: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-0/wal/wals/69d3bf1fecd54e64988f167c370d061c
I20260929 20:59:37.927718 2793 ts_tablet_manager.cc:1939] T 69d3bf1fecd54e64988f167c370d061c P b44f8fae008f4acf83a4a8b7aef1a2de: Deleting consensus metadata
I20260929 20:59:37.928159 2794 ts_tablet_manager.cc:1918] T 69d3bf1fecd54e64988f167c370d061c P 2bc5dec7746f48b99039072a2834da21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 20:59:37.928270 2794 log.cc:1196] T 69d3bf1fecd54e64988f167c370d061c P 2bc5dec7746f48b99039072a2834da21: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790715556572573-388-0/minicluster-data/ts-2/wal/wals/69d3bf1fecd54e64988f167c370d061c
I20260929 20:59:37.928618 2127 catalog_manager.cc:4841] TS b44f8fae008f4acf83a4a8b7aef1a2de (127.0.97.1:45601): tablet 69d3bf1fecd54e64988f167c370d061c (table test-table [id=6d2c564ff1cd47bdbb0f45ddc779b1a3]) successfully deleted
I20260929 20:59:37.928707 2794 ts_tablet_manager.cc:1939] T 69d3bf1fecd54e64988f167c370d061c P 2bc5dec7746f48b99039072a2834da21: Deleting consensus metadata
I20260929 20:59:37.929569 2124 catalog_manager.cc:4841] TS 2bc5dec7746f48b99039072a2834da21 (127.0.97.3:46649): tablet 69d3bf1fecd54e64988f167c370d061c (table test-table [id=6d2c564ff1cd47bdbb0f45ddc779b1a3]) successfully deleted
Sep 29 20:59:37 dist-test-slave-1dms krb5kdc[2095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715575, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:59:37.938670 2795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:37.910434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39210 (local address 127.0.97.1:45601)
0929 20:59:37.925236 (+ 14802us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:37.925242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:37.925261 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:59:37.925329 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:37.925334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:37.925414 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:37.925536 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:37.927112 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:37.928299 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:37.929685 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:37.930072 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:37.930165 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:37.933964 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:37.933978 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:37.933993 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:37.934027 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:37.936476 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:37.937107 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:37.937115 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:37.937117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:37.937252 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:37.937539 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:37.937542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:37.937544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:37.937843 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:37.938054 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:37.938241 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 20:59:37.938430 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14668,"thread_start_us":14530,"threads_started":1}
W20260929 20:59:37.939301 2342 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39210
W20260929 20:59:37.939893 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 2211: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:59:37.940073 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 2211
I20260929 20:59:37.961824 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 2351
Sep 29 20:59:37 dist-test-slave-1dms krb5kdc[2095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715575, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:59:37.975951 2797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:37.962524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33574 (local address 127.0.97.2:33557)
0929 20:59:37.963037 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:37.963041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:37.963056 (+ 15us) server_negotiation.cc:407] Connection header received
0929 20:59:37.963106 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:37.963109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:37.963163 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:37.963251 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:37.964775 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:37.965502 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:37.966671 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:37.967066 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:37.967157 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:37.970955 (+ 3798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:37.970967 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:37.970982 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:37.971015 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:37.973787 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:37.974344 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:37.974351 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:37.974353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:37.974467 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:37.974728 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:37.974731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:37.974733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:37.975059 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:37.975268 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:37.975451 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 20:59:37.975598 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":274,"threads_started":1}
W20260929 20:59:37.976557 2481 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33574
W20260929 20:59:37.977126 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 2351: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:59:37.977289 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 2351
I20260929 20:59:38.002378 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 2490
Sep 29 20:59:38 dist-test-slave-1dms krb5kdc[2095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715575, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:59:38.016758 2798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:38.003023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56786 (local address 127.0.97.3:46649)
0929 20:59:38.003551 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:38.003556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:38.003574 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:59:38.003637 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:38.003640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:38.003700 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:38.003809 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:38.005048 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:38.005800 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:38.007003 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:38.007400 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:38.007520 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:38.011586 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:38.011600 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:38.011615 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:38.011646 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:38.014263 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:38.014867 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:38.014874 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:38.014877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:38.014997 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:38.015350 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:38.015357 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:38.015360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:38.015673 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:38.015962 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:38.016219 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 20:59:38.016452 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":245,"threads_started":1}
W20260929 20:59:38.017398 2620 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56786
W20260929 20:59:38.017961 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 2490: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:59:38.018096 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 2490
I20260929 20:59:38.039549 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 2108
I20260929 20:59:38.040613 2180 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:59:38.152979 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 2108
2026-09-29T20:59:38Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8382 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:59:38 dist-test-slave-1dms krb5kdc[2802](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:59:38 dist-test-slave-1dms krb5kdc[2802](info): set up 2 sockets
Sep 29 20:59:38 dist-test-slave-1dms krb5kdc[2802](info): commencing operation
krb5kdc: starting...
W20260929 20:59:39.277232 388 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 20:59:39 dist-test-slave-1dms krb5kdc[2802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:59:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:59:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:59:39.459970 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35757
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35757
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:59:39.769367 2815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:39.769704 2815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:39.769850 2815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:39.776877 2815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:59:39.777012 2815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:39.777110 2815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:39.777207 2815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:59:39.777278 2815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:59:39.786818 2815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:39.788101 2815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:39.789454 2815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:39.801414 2821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:39.801462 2820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:39.801380 2823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:39.806212 2815 server_base.cc:1034] running on GCE node
I20260929 20:59:39.807026 2815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:39.808393 2815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:39.809612 2815 hybrid_clock.cc:648] HybridClock initialized: now 1790715579809568 us; error 52 us; skew 500 ppm
I20260929 20:59:39.809953 2815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:59:39 dist-test-slave-1dms krb5kdc[2802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715579, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:59:39.813055 2815 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:59:39.814534 2815 webserver.cc:460] Webserver started at http://127.0.97.62:35063/ using document root <none> and password file <none>
I20260929 20:59:39.815205 2815 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:39.815341 2815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:39.815827 2815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:39.818727 2815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "b22fdebafed54605831bfb19cab0ac55"
format_stamp: "Formatted at 2026-09-29 20:59:39 on dist-test-slave-1dms"
server_key: "43bb3fbf3520650332f8054f918d33a6"
server_key_iv: "adad485ba56b5645c07fccd48d1108b7"
server_key_version: "encryptionkey@0"
I20260929 20:59:39.819494 2815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "b22fdebafed54605831bfb19cab0ac55"
format_stamp: "Formatted at 2026-09-29 20:59:39 on dist-test-slave-1dms"
server_key: "43bb3fbf3520650332f8054f918d33a6"
server_key_iv: "adad485ba56b5645c07fccd48d1108b7"
server_key_version: "encryptionkey@0"
I20260929 20:59:39.824450 2815 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 20:59:39.827813 2830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:39.829300 2815 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:59:39.829473 2815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "b22fdebafed54605831bfb19cab0ac55"
format_stamp: "Formatted at 2026-09-29 20:59:39 on dist-test-slave-1dms"
server_key: "43bb3fbf3520650332f8054f918d33a6"
server_key_iv: "adad485ba56b5645c07fccd48d1108b7"
server_key_version: "encryptionkey@0"
I20260929 20:59:39.829671 2815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:39.867010 2815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:39.870708 2815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:39.870980 2815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:39.889053 2815 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35757
I20260929 20:59:39.889086 2881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35757 every 8 connection(s)
I20260929 20:59:39.890399 2815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:59:39.894001 2882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:39.896149 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 2815
I20260929 20:59:39.896376 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 20:59:39.897001 388 external_mini_cluster.cc:1428] Setting key 699115951f0a4f2918d22f65bba7198c
I20260929 20:59:39.904678 2882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b22fdebafed54605831bfb19cab0ac55: Bootstrap starting.
I20260929 20:59:39.907594 2882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b22fdebafed54605831bfb19cab0ac55: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:39.908617 2882 log.cc:824] T 00000000000000000000000000000000 P b22fdebafed54605831bfb19cab0ac55: Log is configured to *not* fsync() on all Append() calls
Sep 29 20:59:39 dist-test-slave-1dms krb5kdc[2802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:59:39.911759 2882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b22fdebafed54605831bfb19cab0ac55: No bootstrap required, opened a new log
I20260929 20:59:39.918682 2885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:39.899012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59526 (local address 127.0.97.62:35757)
0929 20:59:39.899785 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:39.899797 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:39.899835 (+ 38us) server_negotiation.cc:407] Connection header received
0929 20:59:39.901549 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:39.901572 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:39.901960 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:39.902363 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:39.903782 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:39.905186 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:39.906404 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:39.907033 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:39.907224 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:39.911858 (+ 4634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:39.911870 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:39.911887 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:39.911925 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:39.914966 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:39.915691 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:39.915697 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:39.915702 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:39.915844 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:39.916229 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:39.916232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:39.916234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:39.916717 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:39.916959 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:39.917347 (+ 388us) server_negotiation.cc:299] Negotiation successful
0929 20:59:39.917652 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":365,"threads_started":1}
I20260929 20:59:39.920348 2882 raft_consensus.cc:348] T 00000000000000000000000000000000 P b22fdebafed54605831bfb19cab0ac55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22fdebafed54605831bfb19cab0ac55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35757 } }
I20260929 20:59:39.920676 2882 raft_consensus.cc:374] T 00000000000000000000000000000000 P b22fdebafed54605831bfb19cab0ac55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:39.920785 2882 raft_consensus.cc:729] T 00000000000000000000000000000000 P b22fdebafed54605831bfb19cab0ac55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22fdebafed54605831bfb19cab0ac55, State: Initialized, Role: FOLLOWER
I20260929 20:59:39.921329 2882 consensus_queue.cc:260] T 00000000000000000000000000000000 P b22fdebafed54605831bfb19cab0ac55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22fdebafed54605831bfb19cab0ac55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35757 } }
I20260929 20:59:39.921545 2882 raft_consensus.cc:388] T 00000000000000000000000000000000 P b22fdebafed54605831bfb19cab0ac55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:59:39.921685 2882 raft_consensus.cc:482] T 00000000000000000000000000000000 P b22fdebafed54605831bfb19cab0ac55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:59:39.921846 2882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b22fdebafed54605831bfb19cab0ac55 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:39.923823 2882 raft_consensus.cc:504] T 00000000000000000000000000000000 P b22fdebafed54605831bfb19cab0ac55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22fdebafed54605831bfb19cab0ac55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35757 } }
I20260929 20:59:39.924291 2882 leader_election.cc:302] T 00000000000000000000000000000000 P b22fdebafed54605831bfb19cab0ac55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b22fdebafed54605831bfb19cab0ac55; no voters:
I20260929 20:59:39.924942 2882 leader_election.cc:288] T 00000000000000000000000000000000 P b22fdebafed54605831bfb19cab0ac55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:59:39.925077 2887 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b22fdebafed54605831bfb19cab0ac55 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:59:39.926163 2887 raft_consensus.cc:686] T 00000000000000000000000000000000 P b22fdebafed54605831bfb19cab0ac55 [term 1 LEADER]: Becoming Leader. State: Replica: b22fdebafed54605831bfb19cab0ac55, State: Running, Role: LEADER
I20260929 20:59:39.926673 2887 consensus_queue.cc:237] T 00000000000000000000000000000000 P b22fdebafed54605831bfb19cab0ac55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22fdebafed54605831bfb19cab0ac55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35757 } }
I20260929 20:59:39.926992 2882 sys_catalog.cc:564] T 00000000000000000000000000000000 P b22fdebafed54605831bfb19cab0ac55 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:59:39.929342 2888 sys_catalog.cc:455] T 00000000000000000000000000000000 P b22fdebafed54605831bfb19cab0ac55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b22fdebafed54605831bfb19cab0ac55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22fdebafed54605831bfb19cab0ac55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35757 } } }
I20260929 20:59:39.929677 2888 sys_catalog.cc:458] T 00000000000000000000000000000000 P b22fdebafed54605831bfb19cab0ac55 [sys.catalog]: This master's current role is: LEADER
I20260929 20:59:39.929421 2889 sys_catalog.cc:455] T 00000000000000000000000000000000 P b22fdebafed54605831bfb19cab0ac55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b22fdebafed54605831bfb19cab0ac55. Latest consensus state: current_term: 1 leader_uuid: "b22fdebafed54605831bfb19cab0ac55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22fdebafed54605831bfb19cab0ac55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35757 } } }
I20260929 20:59:39.929821 2889 sys_catalog.cc:458] T 00000000000000000000000000000000 P b22fdebafed54605831bfb19cab0ac55 [sys.catalog]: This master's current role is: LEADER
I20260929 20:59:39.930449 2893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:59:39.935141 2893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:59:39.944304 2893 catalog_manager.cc:1347] Generated new cluster ID: 5dd38376a6de4298bde1a15807358b45
I20260929 20:59:39.944411 2893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:59:39.986264 2893 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:59:39.987596 2893 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:59:39.994316 2893 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b22fdebafed54605831bfb19cab0ac55: Generated new TSK 0
I20260929 20:59:39.995158 2893 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab.
W20260929 20:59:40.549995 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.1: real 0.542s user 0.001s sys 0.009s
I20260929 20:59:40.550971 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35757
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:59:40.859627 2908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:40.859885 2908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:40.859977 2908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:40.867086 2908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:40.867184 2908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:40.867300 2908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:59:40.876389 2908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:40.877592 2908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:40.878804 2908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:40.891038 2916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:40.895989 2913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:40.891011 2914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:40.897464 2908 server_base.cc:1034] running on GCE node
I20260929 20:59:40.898121 2908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:40.899161 2908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:40.900383 2908 hybrid_clock.cc:648] HybridClock initialized: now 1790715580900335 us; error 39 us; skew 500 ppm
I20260929 20:59:40.900712 2908 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:59:40 dist-test-slave-1dms krb5kdc[2802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:59:40.904237 2908 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:59:40.908394 2908 webserver.cc:460] Webserver started at http://127.0.97.1:32945/ using document root <none> and password file <none>
I20260929 20:59:40.909128 2908 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:40.909255 2908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:40.909557 2908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:40.912312 2908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-0/data/instance:
uuid: "31cdbde0507549afab7eb873794ee299"
format_stamp: "Formatted at 2026-09-29 20:59:40 on dist-test-slave-1dms"
server_key: "10829b9d1447380b5d7e66cd922f94c8"
server_key_iv: "eda04ffaa66f6cd823a0cf7aee80d691"
server_key_version: "encryptionkey@0"
I20260929 20:59:40.913089 2908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-0/wal/instance:
uuid: "31cdbde0507549afab7eb873794ee299"
format_stamp: "Formatted at 2026-09-29 20:59:40 on dist-test-slave-1dms"
server_key: "10829b9d1447380b5d7e66cd922f94c8"
server_key_iv: "eda04ffaa66f6cd823a0cf7aee80d691"
server_key_version: "encryptionkey@0"
I20260929 20:59:40.917970 2908 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 20:59:40.921345 2923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:40.922577 2908 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 20:59:40.922842 2908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-0/wal
uuid: "31cdbde0507549afab7eb873794ee299"
format_stamp: "Formatted at 2026-09-29 20:59:40 on dist-test-slave-1dms"
server_key: "10829b9d1447380b5d7e66cd922f94c8"
server_key_iv: "eda04ffaa66f6cd823a0cf7aee80d691"
server_key_version: "encryptionkey@0"
I20260929 20:59:40.923039 2908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:40.946455 2908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:40.950619 2908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:40.950903 2908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:40.951815 2908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:59:40.953097 2908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:59:40.953207 2908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:40.953342 2908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:59:40.953415 2908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:40.985064 2908 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42845
I20260929 20:59:40.985139 3035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42845 every 8 connection(s)
I20260929 20:59:40.986289 2908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:59:40.987447 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 2908
I20260929 20:59:40.987653 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:59:40.988446 388 external_mini_cluster.cc:1428] Setting key 3aa8b1b73e6d122177544ce7b805bee2
Sep 29 20:59:40 dist-test-slave-1dms krb5kdc[2802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 20:59:41.007275 3038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:40.989066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54653 (local address 127.0.97.62:35757)
0929 20:59:40.989552 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:40.989558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:40.990672 (+ 1114us) server_negotiation.cc:407] Connection header received
0929 20:59:40.992429 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:40.992434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:40.992531 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:40.992639 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:40.994102 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:40.995109 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:40.996651 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:40.997473 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:40.997611 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:41.000756 (+ 3145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:41.000765 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:41.000773 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:41.000815 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:41.004420 (+ 3605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:41.005133 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:41.005140 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:41.005145 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:41.005312 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:41.005724 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:41.005729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:41.005732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:41.006082 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:41.006263 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:41.006800 (+ 537us) server_negotiation.cc:299] Negotiation successful
0929 20:59:41.006951 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":258,"threads_started":1}
I20260929 20:59:41.008153 3039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:40.989652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35757 (local address 127.0.97.1:54653)
0929 20:59:40.990515 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:40.990568 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:40.992183 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:40.992915 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:40.992930 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:40.993408 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:40.993941 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:40.993957 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:40.995257 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:40.995260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:40.996511 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:40.996523 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:40.997672 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:40.997677 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:40.997863 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:40.998511 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:59:40.998533 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:41.000540 (+ 2007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:41.004549 (+ 4009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:41.004556 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:41.004567 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:41.005025 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:41.005423 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:41.005426 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:41.005428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:41.005596 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:41.006199 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:41.006204 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:41.006481 (+ 277us) client_negotiation.cc:769] Sending connection context
0929 20:59:41.006693 (+ 212us) client_negotiation.cc:240] Negotiation successful
0929 20:59:41.007041 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":290,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 20:59:41.009716 3036 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35757
I20260929 20:59:41.010028 3036 heartbeater.cc:464] Registering TS with master...
I20260929 20:59:41.010880 3036 heartbeater.cc:511] Master 127.0.97.62:35757 requested a full tablet report, sending...
I20260929 20:59:41.013060 2847 ts_manager.cc:194] Registered new tserver with Master: 31cdbde0507549afab7eb873794ee299 (127.0.97.1:42845)
I20260929 20:59:41.014515 2847 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54653
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:59:41.031661 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35757
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:59:41.342327 3042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:41.342594 3042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:41.342684 3042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:41.349704 3042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:41.349803 3042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:41.349920 3042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:59:41.359326 3042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:41.360479 3042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:41.361696 3042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:41.373957 3047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:41.373927 3050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:41.373914 3048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:41.374799 3042 server_base.cc:1034] running on GCE node
I20260929 20:59:41.375381 3042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:41.376296 3042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:41.377480 3042 hybrid_clock.cc:648] HybridClock initialized: now 1790715581377430 us; error 40 us; skew 500 ppm
I20260929 20:59:41.377779 3042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:59:41 dist-test-slave-1dms krb5kdc[2802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:59:41.381078 3042 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:59:41.382690 3042 webserver.cc:460] Webserver started at http://127.0.97.2:46021/ using document root <none> and password file <none>
I20260929 20:59:41.383370 3042 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:41.383510 3042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:41.383819 3042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:41.386631 3042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-1/data/instance:
uuid: "765075aa3b8948c59bcd46a8882e2f96"
format_stamp: "Formatted at 2026-09-29 20:59:41 on dist-test-slave-1dms"
server_key: "bc47bc33c75de2e1f623f14e410135dc"
server_key_iv: "9b07eaebd260dca4c10b7e168767deb6"
server_key_version: "encryptionkey@0"
I20260929 20:59:41.387377 3042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-1/wal/instance:
uuid: "765075aa3b8948c59bcd46a8882e2f96"
format_stamp: "Formatted at 2026-09-29 20:59:41 on dist-test-slave-1dms"
server_key: "bc47bc33c75de2e1f623f14e410135dc"
server_key_iv: "9b07eaebd260dca4c10b7e168767deb6"
server_key_version: "encryptionkey@0"
I20260929 20:59:41.392205 3042 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 20:59:41.395443 3057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:41.396762 3042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:59:41.396981 3042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-1/wal
uuid: "765075aa3b8948c59bcd46a8882e2f96"
format_stamp: "Formatted at 2026-09-29 20:59:41 on dist-test-slave-1dms"
server_key: "bc47bc33c75de2e1f623f14e410135dc"
server_key_iv: "9b07eaebd260dca4c10b7e168767deb6"
server_key_version: "encryptionkey@0"
I20260929 20:59:41.397183 3042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:41.415479 3042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:41.419610 3042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:41.419898 3042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:41.420822 3042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:59:41.422165 3042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:59:41.422262 3042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:41.422403 3042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:59:41.422472 3042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:41.454169 3042 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45385
I20260929 20:59:41.454193 3169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45385 every 8 connection(s)
I20260929 20:59:41.455577 3042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:59:41.457530 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 3042
I20260929 20:59:41.457748 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:59:41.458248 388 external_mini_cluster.cc:1428] Setting key 966d9619ed77c8cbdc09db646b2b1ff6
Sep 29 20:59:41 dist-test-slave-1dms krb5kdc[2802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 20:59:41.476948 3038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:41.458230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47465 (local address 127.0.97.62:35757)
0929 20:59:41.458455 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:41.458461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:41.459830 (+ 1369us) server_negotiation.cc:407] Connection header received
0929 20:59:41.461768 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:41.461776 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:41.461866 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:41.461978 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:41.463327 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:41.464485 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:41.466923 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:41.467432 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:41.467531 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:41.471467 (+ 3936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:41.471472 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:41.471475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:41.471508 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:41.473998 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:41.474678 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:41.474683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:41.474691 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:41.474825 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:41.475231 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:41.475235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:41.475237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:41.475491 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:41.475638 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:41.476299 (+ 661us) server_negotiation.cc:299] Negotiation successful
0929 20:59:41.476512 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 20:59:41.477914 3172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:41.458813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35757 (local address 127.0.97.2:47465)
0929 20:59:41.459660 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:41.459716 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:41.461510 (+ 1794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:41.462136 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:41.462144 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:41.462630 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:41.463151 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:41.463178 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:41.464643 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:41.464646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:41.466769 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:41.466786 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:41.467641 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:41.467651 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:41.467966 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:41.468668 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:59:41.468700 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:41.471260 (+ 2560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:41.474130 (+ 2870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:41.474139 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:41.474155 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:41.474554 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:41.474930 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:41.474936 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:41.474940 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:41.475113 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:41.475611 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:41.475619 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:41.475967 (+ 348us) client_negotiation.cc:769] Sending connection context
0929 20:59:41.476217 (+ 250us) client_negotiation.cc:240] Negotiation successful
0929 20:59:41.476711 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":365,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 20:59:41.479501 3170 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35757
I20260929 20:59:41.479816 3170 heartbeater.cc:464] Registering TS with master...
I20260929 20:59:41.480504 3170 heartbeater.cc:511] Master 127.0.97.62:35757 requested a full tablet report, sending...
I20260929 20:59:41.481942 2847 ts_manager.cc:194] Registered new tserver with Master: 765075aa3b8948c59bcd46a8882e2f96 (127.0.97.2:45385)
I20260929 20:59:41.482687 2847 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47465
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:59:41.500615 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35757
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:59:41.812999 3175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:41.813294 3175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:41.813447 3175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:41.820420 3175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:41.820554 3175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:41.820730 3175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:59:41.830427 3175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:41.831691 3175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:41.832999 3175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:41.845882 3180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:41.845942 3181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:41.846088 3183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:41.846697 3175 server_base.cc:1034] running on GCE node
I20260929 20:59:41.847198 3175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:41.848006 3175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:41.849192 3175 hybrid_clock.cc:648] HybridClock initialized: now 1790715581849152 us; error 35 us; skew 500 ppm
I20260929 20:59:41.849448 3175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:59:41 dist-test-slave-1dms krb5kdc[2802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:59:41.852361 3175 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:59:41.853765 3175 webserver.cc:460] Webserver started at http://127.0.97.3:39029/ using document root <none> and password file <none>
I20260929 20:59:41.854373 3175 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:41.854470 3175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:41.854703 3175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:41.857445 3175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-2/data/instance:
uuid: "3780fab9ec7a4ab78ab7bae1f2e2d4a9"
format_stamp: "Formatted at 2026-09-29 20:59:41 on dist-test-slave-1dms"
server_key: "cbeb70adb510cbad935d542d79af89e9"
server_key_iv: "f1d64838ec9b2d4c62026e25d0cc90c3"
server_key_version: "encryptionkey@0"
I20260929 20:59:41.858125 3175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-2/wal/instance:
uuid: "3780fab9ec7a4ab78ab7bae1f2e2d4a9"
format_stamp: "Formatted at 2026-09-29 20:59:41 on dist-test-slave-1dms"
server_key: "cbeb70adb510cbad935d542d79af89e9"
server_key_iv: "f1d64838ec9b2d4c62026e25d0cc90c3"
server_key_version: "encryptionkey@0"
I20260929 20:59:41.863258 3175 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:59:41.866582 3190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:41.867826 3175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:59:41.868018 3175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-2/wal
uuid: "3780fab9ec7a4ab78ab7bae1f2e2d4a9"
format_stamp: "Formatted at 2026-09-29 20:59:41 on dist-test-slave-1dms"
server_key: "cbeb70adb510cbad935d542d79af89e9"
server_key_iv: "f1d64838ec9b2d4c62026e25d0cc90c3"
server_key_version: "encryptionkey@0"
I20260929 20:59:41.868244 3175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:41.894006 3175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:41.898284 3175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:41.898556 3175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:41.899451 3175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:59:41.900787 3175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:59:41.900909 3175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260929 20:59:41.901036 3175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:59:41.901101 3175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:41.931770 3175 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43703
I20260929 20:59:41.931851 3302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43703 every 8 connection(s)
I20260929 20:59:41.933147 3175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:59:41.936432 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 3175
I20260929 20:59:41.936662 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:59:41.937227 388 external_mini_cluster.cc:1428] Setting key e1c15a879f3ae187b9777e075385a3c3
Sep 29 20:59:41 dist-test-slave-1dms krb5kdc[2802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:59:41.953109 3038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:41.935588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42925 (local address 127.0.97.62:35757)
0929 20:59:41.935799 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:41.935803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:41.936968 (+ 1165us) server_negotiation.cc:407] Connection header received
0929 20:59:41.939213 (+ 2245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:41.939218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:41.939296 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:41.939382 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:41.941169 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:41.941906 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:41.943422 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:41.943839 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:41.943940 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:41.947225 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:41.947230 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:41.947234 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:41.947267 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:41.949815 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:41.950505 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:41.950511 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:41.950514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:41.950655 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:41.951146 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:41.951151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:41.951153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:41.951505 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:41.951652 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:41.952550 (+ 898us) server_negotiation.cc:299] Negotiation successful
0929 20:59:41.952794 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 20:59:41.954154 3305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:41.935985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35757 (local address 127.0.97.3:42925)
0929 20:59:41.936793 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:41.936864 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:41.938899 (+ 2035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:41.939650 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:41.939664 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:41.940369 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:41.941001 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:41.941020 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:41.942048 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:41.942051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:41.943300 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:41.943315 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:41.944031 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:41.944036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:41.944240 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:41.944926 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:59:41.944949 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:41.947031 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:41.949944 (+ 2913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:41.949951 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:41.949963 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:41.950382 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:41.950796 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:41.950800 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:41.950803 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:41.951036 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:41.951671 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:41.951677 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:41.952170 (+ 493us) client_negotiation.cc:769] Sending connection context
0929 20:59:41.952448 (+ 278us) client_negotiation.cc:240] Negotiation successful
0929 20:59:41.952958 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":361,"threads_started":1}
I20260929 20:59:41.955565 3303 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35757
I20260929 20:59:41.955893 3303 heartbeater.cc:464] Registering TS with master...
I20260929 20:59:41.956554 3303 heartbeater.cc:511] Master 127.0.97.62:35757 requested a full tablet report, sending...
I20260929 20:59:41.958009 2847 ts_manager.cc:194] Registered new tserver with Master: 3780fab9ec7a4ab78ab7bae1f2e2d4a9 (127.0.97.3:43703)
I20260929 20:59:41.958796 2847 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42925
I20260929 20:59:41.964565 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 20:59:41.994088 3038 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:59:41.991731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59536 (local address 127.0.97.62:35757)
0929 20:59:41.991910 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:41.991918 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:41.992362 (+ 444us) server_negotiation.cc:407] Connection header received
0929 20:59:41.992569 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:41.992573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:41.992631 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:41.992731 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:41.993920 (+ 1189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59536: BlockingRecv error: recv got EOF from 127.0.0.1:59536 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 20:59:41.997278 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 2908
W20260929 20:59:41.999912 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 2908: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:42845: server requires authentication, but client does not have Kerberos credentials available
I20260929 20:59:42.000087 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 2908
I20260929 20:59:42.015084 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 3042
W20260929 20:59:42.017464 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 3042: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:45385: server requires authentication, but client does not have Kerberos credentials available
I20260929 20:59:42.017624 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 3042
I20260929 20:59:42.032415 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 3175
W20260929 20:59:42.035112 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 3175: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:43703: server requires authentication, but client does not have Kerberos credentials available
I20260929 20:59:42.035243 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 3175
I20260929 20:59:42.050235 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 2815
I20260929 20:59:42.051316 2877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:59:42.151202 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 2815
2026-09-29T20:59:42Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3995 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:59:42 dist-test-slave-1dms krb5kdc[3320](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:59:42 dist-test-slave-1dms krb5kdc[3320](info): set up 2 sockets
Sep 29 20:59:42 dist-test-slave-1dms krb5kdc[3320](info): commencing operation
krb5kdc: starting...
W20260929 20:59:43.245154 388 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.002s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 20:59:43 dist-test-slave-1dms krb5kdc[3320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:59:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:59:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:59:43.432518 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46339
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:59:43.745010 3333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:43.745297 3333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:43.745464 3333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:43.753106 3333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:59:43.753213 3333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:43.753355 3333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:43.753433 3333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:59:43.753494 3333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:59:43.763749 3333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:43.765125 3333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:43.766446 3333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:43.779049 3339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:43.779139 3341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:43.779309 3338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:43.779470 3333 server_base.cc:1034] running on GCE node
I20260929 20:59:43.780534 3333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:43.782186 3333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:43.783439 3333 hybrid_clock.cc:648] HybridClock initialized: now 1790715583783403 us; error 45 us; skew 500 ppm
I20260929 20:59:43.783847 3333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:59:43 dist-test-slave-1dms krb5kdc[3320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:59:43.787257 3333 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:59:43.788769 3333 webserver.cc:460] Webserver started at http://127.0.97.62:44563/ using document root <none> and password file <none>
I20260929 20:59:43.789494 3333 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:43.789654 3333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:43.789963 3333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:43.793049 3333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "147d103dd4214f188cf9565a80b1b2bb"
format_stamp: "Formatted at 2026-09-29 20:59:43 on dist-test-slave-1dms"
server_key: "62e1c140b6fabb47b1892de94be9bd74"
server_key_iv: "9aab442ee6e0530783f942de219f9d73"
server_key_version: "encryptionkey@0"
I20260929 20:59:43.793913 3333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "147d103dd4214f188cf9565a80b1b2bb"
format_stamp: "Formatted at 2026-09-29 20:59:43 on dist-test-slave-1dms"
server_key: "62e1c140b6fabb47b1892de94be9bd74"
server_key_iv: "9aab442ee6e0530783f942de219f9d73"
server_key_version: "encryptionkey@0"
I20260929 20:59:43.799113 3333 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:59:43.802549 3348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:43.803925 3333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:59:43.804103 3333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "147d103dd4214f188cf9565a80b1b2bb"
format_stamp: "Formatted at 2026-09-29 20:59:43 on dist-test-slave-1dms"
server_key: "62e1c140b6fabb47b1892de94be9bd74"
server_key_iv: "9aab442ee6e0530783f942de219f9d73"
server_key_version: "encryptionkey@0"
I20260929 20:59:43.804297 3333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:43.817149 3333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:43.821372 3333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:43.821667 3333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:43.840212 3333 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46339
I20260929 20:59:43.840235 3399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46339 every 8 connection(s)
I20260929 20:59:43.841893 3333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:59:43.845614 3400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:43.848482 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 3333
I20260929 20:59:43.848683 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 20:59:43.849490 388 external_mini_cluster.cc:1428] Setting key 48cbeb6a9cd0916d9ba307c361c3975e
I20260929 20:59:43.858435 3400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 147d103dd4214f188cf9565a80b1b2bb: Bootstrap starting.
I20260929 20:59:43.862172 3400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 147d103dd4214f188cf9565a80b1b2bb: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:43.863467 3400 log.cc:824] T 00000000000000000000000000000000 P 147d103dd4214f188cf9565a80b1b2bb: Log is configured to *not* fsync() on all Append() calls
Sep 29 20:59:43 dist-test-slave-1dms krb5kdc[3320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:59:43.866392 3400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 147d103dd4214f188cf9565a80b1b2bb: No bootstrap required, opened a new log
I20260929 20:59:43.872556 3403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:43.851451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56460 (local address 127.0.97.62:46339)
0929 20:59:43.852250 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:43.852262 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:43.852301 (+ 39us) server_negotiation.cc:407] Connection header received
0929 20:59:43.853953 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:43.853975 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:43.854394 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:43.854729 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:43.856619 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:43.857966 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:43.859673 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:43.860464 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:43.860600 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:43.865477 (+ 4877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:43.865484 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:43.865499 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:43.865532 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:43.868597 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:43.869329 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:43.869344 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:43.869350 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:43.869507 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:43.869882 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:43.869889 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:43.869892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:43.870415 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:43.870734 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:43.871169 (+ 435us) server_negotiation.cc:299] Negotiation successful
0929 20:59:43.871556 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":377,"threads_started":1}
I20260929 20:59:43.874637 3400 raft_consensus.cc:348] T 00000000000000000000000000000000 P 147d103dd4214f188cf9565a80b1b2bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147d103dd4214f188cf9565a80b1b2bb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46339 } }
I20260929 20:59:43.875058 3400 raft_consensus.cc:374] T 00000000000000000000000000000000 P 147d103dd4214f188cf9565a80b1b2bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:43.875183 3400 raft_consensus.cc:729] T 00000000000000000000000000000000 P 147d103dd4214f188cf9565a80b1b2bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 147d103dd4214f188cf9565a80b1b2bb, State: Initialized, Role: FOLLOWER
I20260929 20:59:43.875801 3400 consensus_queue.cc:260] T 00000000000000000000000000000000 P 147d103dd4214f188cf9565a80b1b2bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147d103dd4214f188cf9565a80b1b2bb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46339 } }
I20260929 20:59:43.876050 3400 raft_consensus.cc:388] T 00000000000000000000000000000000 P 147d103dd4214f188cf9565a80b1b2bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:59:43.876196 3400 raft_consensus.cc:482] T 00000000000000000000000000000000 P 147d103dd4214f188cf9565a80b1b2bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:59:43.876401 3400 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 147d103dd4214f188cf9565a80b1b2bb [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:43.876993 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:59:43.878669 3400 raft_consensus.cc:504] T 00000000000000000000000000000000 P 147d103dd4214f188cf9565a80b1b2bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147d103dd4214f188cf9565a80b1b2bb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46339 } }
I20260929 20:59:43.879180 3400 leader_election.cc:302] T 00000000000000000000000000000000 P 147d103dd4214f188cf9565a80b1b2bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 147d103dd4214f188cf9565a80b1b2bb; no voters:
I20260929 20:59:43.879842 3400 leader_election.cc:288] T 00000000000000000000000000000000 P 147d103dd4214f188cf9565a80b1b2bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:59:43.879930 3405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 147d103dd4214f188cf9565a80b1b2bb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:59:43.881130 3405 raft_consensus.cc:686] T 00000000000000000000000000000000 P 147d103dd4214f188cf9565a80b1b2bb [term 1 LEADER]: Becoming Leader. State: Replica: 147d103dd4214f188cf9565a80b1b2bb, State: Running, Role: LEADER
I20260929 20:59:43.881740 3405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 147d103dd4214f188cf9565a80b1b2bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147d103dd4214f188cf9565a80b1b2bb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46339 } }
I20260929 20:59:43.882193 3400 sys_catalog.cc:564] T 00000000000000000000000000000000 P 147d103dd4214f188cf9565a80b1b2bb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:59:43.884996 3406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 147d103dd4214f188cf9565a80b1b2bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "147d103dd4214f188cf9565a80b1b2bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147d103dd4214f188cf9565a80b1b2bb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46339 } } }
I20260929 20:59:43.885311 3406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 147d103dd4214f188cf9565a80b1b2bb [sys.catalog]: This master's current role is: LEADER
I20260929 20:59:43.889473 3407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 147d103dd4214f188cf9565a80b1b2bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 147d103dd4214f188cf9565a80b1b2bb. Latest consensus state: current_term: 1 leader_uuid: "147d103dd4214f188cf9565a80b1b2bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147d103dd4214f188cf9565a80b1b2bb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46339 } } }
I20260929 20:59:43.889868 3407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 147d103dd4214f188cf9565a80b1b2bb [sys.catalog]: This master's current role is: LEADER
I20260929 20:59:43.891179 3411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:59:43.896318 3411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:59:43.915452 3411 catalog_manager.cc:1347] Generated new cluster ID: 76a68f985b2041c58f3d9b6841b54749
I20260929 20:59:43.915652 3411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:59:43.939337 3411 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:59:43.941879 3411 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:59:43.949887 3411 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 147d103dd4214f188cf9565a80b1b2bb: Generated new TSK 0
I20260929 20:59:43.950842 3411 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:59:43.954217 3403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:43.945136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56462 (local address 127.0.97.62:46339)
0929 20:59:43.945300 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:43.945307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:43.945993 (+ 686us) server_negotiation.cc:407] Connection header received
0929 20:59:43.946157 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:43.946165 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:43.946253 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:43.946381 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:43.947572 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:43.948735 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:43.950508 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:43.951186 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:43.951318 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:43.952527 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:43.952533 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:43.952538 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:59:43.953391 (+ 853us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:43.953464 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:59:43.953497 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:43.953607 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:43.953776 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 20:59:43.953872 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 20:59:43.960531 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 3333
I20260929 20:59:43.962301 3395 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:59:44.050534 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 3333
2026-09-29T20:59:44Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1891 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:59:44 dist-test-slave-1dms krb5kdc[3434](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:59:44 dist-test-slave-1dms krb5kdc[3434](info): set up 2 sockets
Sep 29 20:59:44 dist-test-slave-1dms krb5kdc[3434](info): commencing operation
krb5kdc: starting...
W20260929 20:59:45.121218 388 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 20:59:45 dist-test-slave-1dms krb5kdc[3434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:59:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:59:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:59:45.316438 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42331
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42331
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:59:45.634485 3447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:45.634852 3447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:45.635015 3447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:45.642371 3447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:59:45.642532 3447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:45.642653 3447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:45.642764 3447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:59:45.642841 3447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:59:45.652797 3447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:45.654277 3447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:45.655571 3447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:45.667325 3452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:45.667241 3453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:45.667286 3455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:45.668201 3447 server_base.cc:1034] running on GCE node
I20260929 20:59:45.669149 3447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:45.670702 3447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:45.671901 3447 hybrid_clock.cc:648] HybridClock initialized: now 1790715585671862 us; error 33 us; skew 500 ppm
I20260929 20:59:45.672240 3447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:59:45 dist-test-slave-1dms krb5kdc[3434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:59:45.675557 3447 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:59:45.677074 3447 webserver.cc:460] Webserver started at http://127.0.97.62:42515/ using document root <none> and password file <none>
I20260929 20:59:45.677913 3447 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:45.678058 3447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:45.678377 3447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:45.681577 3447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "07e8f65484cb4c5ab8d3023f0a417d5e"
format_stamp: "Formatted at 2026-09-29 20:59:45 on dist-test-slave-1dms"
server_key: "ee8ececb0416da11feed0a275cc84928"
server_key_iv: "78dfbc063739df9d27ea7518401b9395"
server_key_version: "encryptionkey@0"
I20260929 20:59:45.682340 3447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "07e8f65484cb4c5ab8d3023f0a417d5e"
format_stamp: "Formatted at 2026-09-29 20:59:45 on dist-test-slave-1dms"
server_key: "ee8ececb0416da11feed0a275cc84928"
server_key_iv: "78dfbc063739df9d27ea7518401b9395"
server_key_version: "encryptionkey@0"
I20260929 20:59:45.687901 3447 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260929 20:59:45.691834 3462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:45.693272 3447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:59:45.693482 3447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "07e8f65484cb4c5ab8d3023f0a417d5e"
format_stamp: "Formatted at 2026-09-29 20:59:45 on dist-test-slave-1dms"
server_key: "ee8ececb0416da11feed0a275cc84928"
server_key_iv: "78dfbc063739df9d27ea7518401b9395"
server_key_version: "encryptionkey@0"
I20260929 20:59:45.693728 3447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:45.709733 3447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:45.713985 3447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:45.714258 3447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:45.733167 3447 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42331
I20260929 20:59:45.733209 3513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42331 every 8 connection(s)
I20260929 20:59:45.734611 3447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:59:45.738131 3514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:45.744815 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 3447
I20260929 20:59:45.745049 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 20:59:45.745846 388 external_mini_cluster.cc:1428] Setting key c4a4e4e12e3cf03bd4c7200d76e26302
I20260929 20:59:45.749167 3514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07e8f65484cb4c5ab8d3023f0a417d5e: Bootstrap starting.
I20260929 20:59:45.752110 3514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07e8f65484cb4c5ab8d3023f0a417d5e: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:45.753162 3514 log.cc:824] T 00000000000000000000000000000000 P 07e8f65484cb4c5ab8d3023f0a417d5e: Log is configured to *not* fsync() on all Append() calls
I20260929 20:59:45.755844 3514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07e8f65484cb4c5ab8d3023f0a417d5e: No bootstrap required, opened a new log
Sep 29 20:59:45 dist-test-slave-1dms krb5kdc[3434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:59:45.763752 3514 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07e8f65484cb4c5ab8d3023f0a417d5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e8f65484cb4c5ab8d3023f0a417d5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42331 } }
I20260929 20:59:45.764200 3514 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07e8f65484cb4c5ab8d3023f0a417d5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:45.764303 3514 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07e8f65484cb4c5ab8d3023f0a417d5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07e8f65484cb4c5ab8d3023f0a417d5e, State: Initialized, Role: FOLLOWER
I20260929 20:59:45.765014 3514 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07e8f65484cb4c5ab8d3023f0a417d5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e8f65484cb4c5ab8d3023f0a417d5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42331 } }
I20260929 20:59:45.765260 3514 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07e8f65484cb4c5ab8d3023f0a417d5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:59:45.765408 3514 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07e8f65484cb4c5ab8d3023f0a417d5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:59:45.765607 3514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07e8f65484cb4c5ab8d3023f0a417d5e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:45.768062 3517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:45.747829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54268 (local address 127.0.97.62:42331)
0929 20:59:45.748568 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:45.748579 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:45.748617 (+ 38us) server_negotiation.cc:407] Connection header received
0929 20:59:45.750392 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:45.750413 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:45.750801 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:45.751180 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:45.752614 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:45.754041 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:45.755305 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:45.756027 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:45.756219 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:45.760457 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:45.760465 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:45.760479 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:45.760516 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:45.763636 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:45.764512 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:45.764520 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:45.764535 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:45.764692 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:45.765106 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:45.765113 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:45.765116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:45.765843 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:45.766176 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:45.766639 (+ 463us) server_negotiation.cc:299] Negotiation successful
0929 20:59:45.767053 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":340,"threads_started":1}
I20260929 20:59:45.792246 3514 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07e8f65484cb4c5ab8d3023f0a417d5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e8f65484cb4c5ab8d3023f0a417d5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42331 } }
I20260929 20:59:45.792927 3514 leader_election.cc:302] T 00000000000000000000000000000000 P 07e8f65484cb4c5ab8d3023f0a417d5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07e8f65484cb4c5ab8d3023f0a417d5e; no voters:
I20260929 20:59:45.793516 3514 leader_election.cc:288] T 00000000000000000000000000000000 P 07e8f65484cb4c5ab8d3023f0a417d5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:59:45.793661 3519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07e8f65484cb4c5ab8d3023f0a417d5e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:59:45.794399 3519 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07e8f65484cb4c5ab8d3023f0a417d5e [term 1 LEADER]: Becoming Leader. State: Replica: 07e8f65484cb4c5ab8d3023f0a417d5e, State: Running, Role: LEADER
I20260929 20:59:45.794920 3519 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07e8f65484cb4c5ab8d3023f0a417d5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e8f65484cb4c5ab8d3023f0a417d5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42331 } }
I20260929 20:59:45.795680 3514 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07e8f65484cb4c5ab8d3023f0a417d5e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:59:45.797477 3520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07e8f65484cb4c5ab8d3023f0a417d5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07e8f65484cb4c5ab8d3023f0a417d5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e8f65484cb4c5ab8d3023f0a417d5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42331 } } }
I20260929 20:59:45.797564 3521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07e8f65484cb4c5ab8d3023f0a417d5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07e8f65484cb4c5ab8d3023f0a417d5e. Latest consensus state: current_term: 1 leader_uuid: "07e8f65484cb4c5ab8d3023f0a417d5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e8f65484cb4c5ab8d3023f0a417d5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42331 } } }
I20260929 20:59:45.798332 3520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07e8f65484cb4c5ab8d3023f0a417d5e [sys.catalog]: This master's current role is: LEADER
I20260929 20:59:45.798379 3521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07e8f65484cb4c5ab8d3023f0a417d5e [sys.catalog]: This master's current role is: LEADER
I20260929 20:59:45.799304 3525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:59:45.804569 3525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:59:45.813664 3525 catalog_manager.cc:1347] Generated new cluster ID: 15be63f7bddf4d4ba518e54730265c4a
I20260929 20:59:45.813789 3525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:59:45.830258 3525 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:59:45.831538 3525 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:59:45.837998 3525 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07e8f65484cb4c5ab8d3023f0a417d5e: Generated new TSK 0
I20260929 20:59:45.838970 3525 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:59:45.892678 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42331
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:59:46.213079 3540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:46.213335 3540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:46.213501 3540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:46.220665 3540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:46.220767 3540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:46.220963 3540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:59:46.230260 3540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:46.231487 3540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:46.232782 3540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:46.246330 3545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:46.246381 3548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:46.246411 3546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:46.247262 3540 server_base.cc:1034] running on GCE node
I20260929 20:59:46.247849 3540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:46.248740 3540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:46.249955 3540 hybrid_clock.cc:648] HybridClock initialized: now 1790715586249936 us; error 69 us; skew 500 ppm
I20260929 20:59:46.250254 3540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:59:46 dist-test-slave-1dms krb5kdc[3434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:59:46.258469 3540 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:59:46.259835 3540 webserver.cc:460] Webserver started at http://127.0.97.1:46379/ using document root <none> and password file <none>
I20260929 20:59:46.260558 3540 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:46.260721 3540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:46.261097 3540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:46.263916 3540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-0/data/instance:
uuid: "38a6c7108a864af590ae8dad482191e4"
format_stamp: "Formatted at 2026-09-29 20:59:46 on dist-test-slave-1dms"
server_key: "e3d42ab80f7879bfb87a10d5a8af65c0"
server_key_iv: "45f3d3663222ec4389365871bc29e876"
server_key_version: "encryptionkey@0"
I20260929 20:59:46.264715 3540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-0/wal/instance:
uuid: "38a6c7108a864af590ae8dad482191e4"
format_stamp: "Formatted at 2026-09-29 20:59:46 on dist-test-slave-1dms"
server_key: "e3d42ab80f7879bfb87a10d5a8af65c0"
server_key_iv: "45f3d3663222ec4389365871bc29e876"
server_key_version: "encryptionkey@0"
I20260929 20:59:46.269855 3540 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:59:46.273182 3555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:46.274510 3540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:59:46.274681 3540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-0/wal
uuid: "38a6c7108a864af590ae8dad482191e4"
format_stamp: "Formatted at 2026-09-29 20:59:46 on dist-test-slave-1dms"
server_key: "e3d42ab80f7879bfb87a10d5a8af65c0"
server_key_iv: "45f3d3663222ec4389365871bc29e876"
server_key_version: "encryptionkey@0"
I20260929 20:59:46.274888 3540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:46.292541 3540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:46.297008 3540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:46.297300 3540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:46.298218 3540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:59:46.299713 3540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:59:46.299813 3540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:46.299958 3540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:59:46.300029 3540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:46.334475 3540 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44189
I20260929 20:59:46.334558 3667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44189 every 8 connection(s)
I20260929 20:59:46.336174 3540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:59:46.342264 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 3540
I20260929 20:59:46.342459 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:59:46.343307 388 external_mini_cluster.cc:1428] Setting key c9fe00922552539592503aff82854fea
Sep 29 20:59:46 dist-test-slave-1dms krb5kdc[3434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:59:46.360237 3670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:46.339013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55691 (local address 127.0.97.62:42331)
0929 20:59:46.339423 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:46.339429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:46.340620 (+ 1191us) server_negotiation.cc:407] Connection header received
0929 20:59:46.342467 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:46.342476 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:46.342561 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:46.342718 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:46.344475 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:46.345504 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:46.348261 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:46.348643 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:46.348734 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:46.353415 (+ 4681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:46.353425 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:46.353433 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:46.353476 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:46.357037 (+ 3561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:46.357706 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:46.357714 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:46.357721 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:46.357863 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:46.358311 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:46.358318 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:46.358325 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:46.358693 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:46.358840 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:46.359509 (+ 669us) server_negotiation.cc:299] Negotiation successful
0929 20:59:46.359756 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":192,"threads_started":1}
I20260929 20:59:46.361042 3671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:46.339679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42331 (local address 127.0.97.1:55691)
0929 20:59:46.340444 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:46.340509 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:46.342175 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:46.342895 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:46.342906 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:46.343762 (+ 856us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:46.344344 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:46.344361 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:46.345693 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:46.345700 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:46.348097 (+ 2397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:46.348128 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:46.348896 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:46.348907 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:46.349267 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:46.350020 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:59:46.350058 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:46.353199 (+ 3141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:46.357165 (+ 3966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:46.357171 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:46.357184 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:46.357577 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:46.357993 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:46.357997 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:46.358000 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:46.358189 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:46.358815 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:46.358821 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:46.359134 (+ 313us) client_negotiation.cc:769] Sending connection context
0929 20:59:46.359384 (+ 250us) client_negotiation.cc:240] Negotiation successful
0929 20:59:46.359755 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":264,"threads_started":1}
I20260929 20:59:46.362594 3668 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42331
I20260929 20:59:46.362898 3668 heartbeater.cc:464] Registering TS with master...
I20260929 20:59:46.363615 3668 heartbeater.cc:511] Master 127.0.97.62:42331 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 20:59:46.365808 3479 ts_manager.cc:194] Registered new tserver with Master: 38a6c7108a864af590ae8dad482191e4 (127.0.97.1:44189)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 20:59:46.367425 3479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55691
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:59:46.390352 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42331
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:59:46.713912 3674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:46.714233 3674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:46.714399 3674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:46.721547 3674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:46.721688 3674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:46.721861 3674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:59:46.731194 3674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:46.732474 3674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:46.733857 3674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:46.746345 3680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:46.746843 3679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:46.747044 3682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:46.747408 3674 server_base.cc:1034] running on GCE node
I20260929 20:59:46.748000 3674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:46.748955 3674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:46.750166 3674 hybrid_clock.cc:648] HybridClock initialized: now 1790715586750117 us; error 44 us; skew 500 ppm
I20260929 20:59:46.750469 3674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:59:46 dist-test-slave-1dms krb5kdc[3434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:59:46.753445 3674 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:59:46.754768 3674 webserver.cc:460] Webserver started at http://127.0.97.2:41855/ using document root <none> and password file <none>
I20260929 20:59:46.755436 3674 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:46.755575 3674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:46.755868 3674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:46.758637 3674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-1/data/instance:
uuid: "587db80465ba4b2bae3230ad2ad25b84"
format_stamp: "Formatted at 2026-09-29 20:59:46 on dist-test-slave-1dms"
server_key: "c640c783673aa4115d69d901314dba75"
server_key_iv: "0194d93e93e23aaa03b071a8490db59e"
server_key_version: "encryptionkey@0"
I20260929 20:59:46.759405 3674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-1/wal/instance:
uuid: "587db80465ba4b2bae3230ad2ad25b84"
format_stamp: "Formatted at 2026-09-29 20:59:46 on dist-test-slave-1dms"
server_key: "c640c783673aa4115d69d901314dba75"
server_key_iv: "0194d93e93e23aaa03b071a8490db59e"
server_key_version: "encryptionkey@0"
I20260929 20:59:46.764271 3674 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 20:59:46.767648 3689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:46.768990 3674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:59:46.769155 3674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-1/wal
uuid: "587db80465ba4b2bae3230ad2ad25b84"
format_stamp: "Formatted at 2026-09-29 20:59:46 on dist-test-slave-1dms"
server_key: "c640c783673aa4115d69d901314dba75"
server_key_iv: "0194d93e93e23aaa03b071a8490db59e"
server_key_version: "encryptionkey@0"
I20260929 20:59:46.769343 3674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:46.789808 3674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:46.793689 3674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:46.793985 3674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:46.794932 3674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:59:46.796355 3674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:59:46.796458 3674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:46.796602 3674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:59:46.796674 3674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:46.829382 3674 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44155
I20260929 20:59:46.829457 3801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44155 every 8 connection(s)
I20260929 20:59:46.830610 3674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:59:46.837764 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 3674
I20260929 20:59:46.837951 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:59:46.838400 388 external_mini_cluster.cc:1428] Setting key ec6aeda94d108e3b7743f32b1b67905f
Sep 29 20:59:46 dist-test-slave-1dms krb5kdc[3434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:59:46.850662 3670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:46.833130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45929 (local address 127.0.97.62:42331)
0929 20:59:46.833305 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:46.833310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:46.834486 (+ 1176us) server_negotiation.cc:407] Connection header received
0929 20:59:46.836264 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:46.836270 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:46.836371 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:46.836480 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:46.837962 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:46.838953 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:46.840539 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:46.841228 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:46.841380 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:46.844872 (+ 3492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:46.844882 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:46.844890 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:46.844933 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:46.847525 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:46.848436 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:46.848440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:46.848442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:46.848556 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:46.849053 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:46.849056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:46.849058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:46.849290 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:46.849395 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:46.850277 (+ 882us) server_negotiation.cc:299] Negotiation successful
0929 20:59:46.850429 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:59:46.852226 3804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:46.833637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42331 (local address 127.0.97.2:45929)
0929 20:59:46.834324 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:46.834368 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:46.835991 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:46.836716 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:46.836724 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:46.837238 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:46.837780 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:46.837797 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:46.839093 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:46.839099 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:46.840383 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:46.840400 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:46.841431 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:46.841436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:46.841630 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:46.842335 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:59:46.842360 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:46.844634 (+ 2274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:46.847660 (+ 3026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:46.847671 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:46.847691 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:46.848325 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:46.848664 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:46.848671 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:46.848677 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:46.848938 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:46.849412 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:46.849422 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:46.849883 (+ 461us) client_negotiation.cc:769] Sending connection context
0929 20:59:46.850188 (+ 305us) client_negotiation.cc:240] Negotiation successful
0929 20:59:46.850733 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":270,"threads_started":1}
I20260929 20:59:46.853659 3802 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42331
I20260929 20:59:46.854010 3802 heartbeater.cc:464] Registering TS with master...
I20260929 20:59:46.854725 3802 heartbeater.cc:511] Master 127.0.97.62:42331 requested a full tablet report, sending...
I20260929 20:59:46.856062 3479 ts_manager.cc:194] Registered new tserver with Master: 587db80465ba4b2bae3230ad2ad25b84 (127.0.97.2:44155)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 20:59:46.856935 3479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45929
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:59:46.880308 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42331
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:59:47.194561 3807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:47.194929 3807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:47.195137 3807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:47.202375 3807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:47.202468 3807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:47.202575 3807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:59:47.212122 3807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:47.213338 3807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:47.214532 3807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:47.227216 3812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:47.227286 3813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:47.227316 3815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:47.228297 3807 server_base.cc:1034] running on GCE node
I20260929 20:59:47.228893 3807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:47.229733 3807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:47.230965 3807 hybrid_clock.cc:648] HybridClock initialized: now 1790715587230927 us; error 49 us; skew 500 ppm
I20260929 20:59:47.231251 3807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:59:47 dist-test-slave-1dms krb5kdc[3434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:59:47.234154 3807 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:59:47.235435 3807 webserver.cc:460] Webserver started at http://127.0.97.3:40693/ using document root <none> and password file <none>
I20260929 20:59:47.236068 3807 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:47.236207 3807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:47.236485 3807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:47.239233 3807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-2/data/instance:
uuid: "35703356e4ab42adaf79482dde95f582"
format_stamp: "Formatted at 2026-09-29 20:59:47 on dist-test-slave-1dms"
server_key: "30414cb04fd730659784168e419f9d53"
server_key_iv: "06265cfb37a5fa7f845ea372127b52f5"
server_key_version: "encryptionkey@0"
I20260929 20:59:47.239990 3807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-2/wal/instance:
uuid: "35703356e4ab42adaf79482dde95f582"
format_stamp: "Formatted at 2026-09-29 20:59:47 on dist-test-slave-1dms"
server_key: "30414cb04fd730659784168e419f9d53"
server_key_iv: "06265cfb37a5fa7f845ea372127b52f5"
server_key_version: "encryptionkey@0"
I20260929 20:59:47.244789 3807 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 20:59:47.247957 3822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:47.249184 3807 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 20:59:47.249356 3807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-2/wal
uuid: "35703356e4ab42adaf79482dde95f582"
format_stamp: "Formatted at 2026-09-29 20:59:47 on dist-test-slave-1dms"
server_key: "30414cb04fd730659784168e419f9d53"
server_key_iv: "06265cfb37a5fa7f845ea372127b52f5"
server_key_version: "encryptionkey@0"
I20260929 20:59:47.249537 3807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:47.275409 3807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:47.279778 3807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:47.280082 3807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:47.281109 3807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:59:47.282418 3807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:59:47.282521 3807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:47.282675 3807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:59:47.282744 3807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:47.315826 3807 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42935
I20260929 20:59:47.315876 3934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42935 every 8 connection(s)
I20260929 20:59:47.317157 3807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:59:47.327431 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 3807
I20260929 20:59:47.327602 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790715556572573-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:59:47.328058 388 external_mini_cluster.cc:1428] Setting key 1a6b669a65fd1a4fbdae3ca46bb5b779
Sep 29 20:59:47 dist-test-slave-1dms krb5kdc[3434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:59:47.337349 3670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:47.319660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60589 (local address 127.0.97.62:42331)
0929 20:59:47.319869 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:47.319875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:47.321048 (+ 1173us) server_negotiation.cc:407] Connection header received
0929 20:59:47.322757 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:47.322761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:47.322827 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:47.322919 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:47.324330 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:47.325156 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:47.326655 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:47.327087 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:47.327219 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:47.331056 (+ 3837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:47.331067 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:47.331070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:47.331100 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:47.334143 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:47.335039 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:47.335043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:47.335045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:47.335166 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:47.335623 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:47.335626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:47.335628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:47.335881 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:47.335988 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:47.336792 (+ 804us) server_negotiation.cc:299] Negotiation successful
0929 20:59:47.337025 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 20:59:47.338528 3937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:47.320137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42331 (local address 127.0.97.3:60589)
0929 20:59:47.320883 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:47.320932 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:47.322501 (+ 1569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:47.323122 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:47.323130 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:47.323613 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:47.324164 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:47.324182 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:47.325298 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:47.325302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:47.326524 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:47.326535 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:47.327283 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:47.327287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:47.327469 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:47.328260 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:59:47.328282 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:47.330858 (+ 2576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:47.334279 (+ 3421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:47.334287 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:47.334304 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:47.334911 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:47.335273 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:47.335278 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:47.335281 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:47.335514 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:47.336000 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:47.336008 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:47.336416 (+ 408us) client_negotiation.cc:769] Sending connection context
0929 20:59:47.336716 (+ 300us) client_negotiation.cc:240] Negotiation successful
0929 20:59:47.337245 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":266,"threads_started":1}
I20260929 20:59:47.340557 3935 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42331
I20260929 20:59:47.340827 3935 heartbeater.cc:464] Registering TS with master...
I20260929 20:59:47.341542 3935 heartbeater.cc:511] Master 127.0.97.62:42331 requested a full tablet report, sending...
I20260929 20:59:47.343125 3479 ts_manager.cc:194] Registered new tserver with Master: 35703356e4ab42adaf79482dde95f582 (127.0.97.3:42935)
I20260929 20:59:47.343987 3479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60589
I20260929 20:59:47.355474 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:59:47.370400 3668 heartbeater.cc:503] Master 127.0.97.62:42331 was elected leader, sending a full tablet report...
Sep 29 20:59:47 dist-test-slave-1dms krb5kdc[3434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715587, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 20:59:47 dist-test-slave-1dms krb5kdc[3434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715587, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:59:47.400471 3670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:47.387513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54278 (local address 127.0.97.62:42331)
0929 20:59:47.387678 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:47.387682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:47.388183 (+ 501us) server_negotiation.cc:407] Connection header received
0929 20:59:47.388322 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:47.388325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:47.388385 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:47.388471 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:47.389830 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:47.390518 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:47.391731 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:47.392134 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:47.392227 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:47.395698 (+ 3471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:47.395703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:47.395706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:47.395737 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:47.398373 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:47.398973 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:47.398976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:47.398979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:47.399095 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:47.399409 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:47.399412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:47.399413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:47.399703 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:47.399878 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:47.400103 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 20:59:47.400259 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260929 20:59:47.401006 3479 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54278
I20260929 20:59:47.403159 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 3540
Sep 29 20:59:47 dist-test-slave-1dms krb5kdc[3434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715587, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:59:47.418388 3946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:47.404136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53372 (local address 127.0.97.1:44189)
0929 20:59:47.404756 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:47.404763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:47.404787 (+ 24us) server_negotiation.cc:407] Connection header received
0929 20:59:47.404880 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:47.404890 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:47.405126 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:47.405250 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:47.406399 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:47.407351 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:47.408461 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:47.409075 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:47.409172 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:47.412959 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:47.412970 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:47.412983 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:47.413011 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:47.415727 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:47.416365 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:47.416372 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:47.416374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:47.416488 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:47.416877 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:47.416883 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:47.416889 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:47.417204 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:47.417501 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:47.417774 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 20:59:47.418045 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":324,"threads_started":1}
W20260929 20:59:47.419616 3663 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53372
W20260929 20:59:47.420691 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 3540: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:59:47.420948 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 3540
I20260929 20:59:47.436817 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 3674
Sep 29 20:59:47 dist-test-slave-1dms krb5kdc[3434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715587, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:59:47.451881 3947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:47.437684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35760 (local address 127.0.97.2:44155)
0929 20:59:47.438241 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:47.438247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:47.438273 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:59:47.438330 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:47.438335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:47.438543 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:47.438670 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:47.439780 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:47.440744 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:47.441995 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:47.442531 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:47.442629 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:47.446660 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:47.446672 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:47.446686 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:47.446731 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:47.449568 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:47.450226 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:47.450232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:47.450234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:47.450348 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:47.450637 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:47.450640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:47.450642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:47.450952 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:47.451151 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:47.451400 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 20:59:47.451582 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":314,"threads_started":1}
W20260929 20:59:47.453222 3797 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35760
W20260929 20:59:47.454352 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 3674: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:59:47.454528 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 3674
I20260929 20:59:47.469751 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 3807
Sep 29 20:59:47 dist-test-slave-1dms krb5kdc[3434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715587, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:59:47.484594 3948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:47.470645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58906 (local address 127.0.97.3:42935)
0929 20:59:47.471192 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:47.471201 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:47.471228 (+ 27us) server_negotiation.cc:407] Connection header received
0929 20:59:47.471284 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:47.471291 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:47.471494 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:47.471614 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:47.472811 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:47.473844 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:47.475022 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:47.475530 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:47.475621 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:47.479632 (+ 4011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:47.479643 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:47.479656 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:47.479689 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:47.482336 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:47.483000 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:47.483006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:47.483008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:47.483117 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:47.483401 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:47.483404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:47.483406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:47.483704 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:47.483920 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:47.484139 (+ 219us) server_negotiation.cc:299] Negotiation successful
0929 20:59:47.484328 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":260,"threads_started":1}
W20260929 20:59:47.485849 3930 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58906
W20260929 20:59:47.486912 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 3807: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:59:47.487077 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 3807
I20260929 20:59:47.502473 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 3447
I20260929 20:59:47.503544 3509 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:59:47.603941 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 3447
2026-09-29T20:59:47Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3561 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:59:47 dist-test-slave-1dms krb5kdc[3952](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:59:47 dist-test-slave-1dms krb5kdc[3952](info): set up 2 sockets
Sep 29 20:59:47 dist-test-slave-1dms krb5kdc[3952](info): commencing operation
krb5kdc: starting...
W20260929 20:59:48.689227 388 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 20:59:48 dist-test-slave-1dms krb5kdc[3952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:59:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:59:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:59:48.880743 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39881
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:59:49.190204 3965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:49.190521 3965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:49.190687 3965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:49.197722 3965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:59:49.197849 3965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:49.197955 3965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:49.198052 3965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:59:49.198144 3965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:59:49.207509 3965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:49.208730 3965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:49.209995 3965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:49.221081 3970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:49.221057 3971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:49.221115 3973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:49.221865 3965 server_base.cc:1034] running on GCE node
I20260929 20:59:49.222565 3965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:49.224346 3965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:49.225558 3965 hybrid_clock.cc:648] HybridClock initialized: now 1790715589225515 us; error 30 us; skew 500 ppm
I20260929 20:59:49.225906 3965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:59:49 dist-test-slave-1dms krb5kdc[3952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:59:49.228737 3965 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:59:49.230288 3965 webserver.cc:460] Webserver started at http://127.0.97.62:43579/ using document root <none> and password file <none>
I20260929 20:59:49.230873 3965 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:49.230965 3965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:49.231204 3965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:49.234020 3965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "647a4ef83d964adc9c6b66780d2be32f"
format_stamp: "Formatted at 2026-09-29 20:59:49 on dist-test-slave-1dms"
server_key: "b4d87fe13ae1eea97b1306539500f435"
server_key_iv: "60453671cce3019f26032f59353051c5"
server_key_version: "encryptionkey@0"
I20260929 20:59:49.234697 3965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "647a4ef83d964adc9c6b66780d2be32f"
format_stamp: "Formatted at 2026-09-29 20:59:49 on dist-test-slave-1dms"
server_key: "b4d87fe13ae1eea97b1306539500f435"
server_key_iv: "60453671cce3019f26032f59353051c5"
server_key_version: "encryptionkey@0"
I20260929 20:59:49.239383 3965 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 20:59:49.242471 3980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:49.243806 3965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:59:49.243942 3965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "647a4ef83d964adc9c6b66780d2be32f"
format_stamp: "Formatted at 2026-09-29 20:59:49 on dist-test-slave-1dms"
server_key: "b4d87fe13ae1eea97b1306539500f435"
server_key_iv: "60453671cce3019f26032f59353051c5"
server_key_version: "encryptionkey@0"
I20260929 20:59:49.244105 3965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:49.257304 3965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:49.261137 3965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:49.261361 3965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:49.279297 3965 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39881
I20260929 20:59:49.279337 4031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39881 every 8 connection(s)
I20260929 20:59:49.280634 3965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:59:49.284241 4032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:49.287493 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 3965
I20260929 20:59:49.287739 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 20:59:49.288390 388 external_mini_cluster.cc:1428] Setting key 9ef255cb10cbc48351392c79bf2ade1f
I20260929 20:59:49.296573 4032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 647a4ef83d964adc9c6b66780d2be32f: Bootstrap starting.
I20260929 20:59:49.300179 4032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 647a4ef83d964adc9c6b66780d2be32f: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:49.301491 4032 log.cc:824] T 00000000000000000000000000000000 P 647a4ef83d964adc9c6b66780d2be32f: Log is configured to *not* fsync() on all Append() calls
Sep 29 20:59:49 dist-test-slave-1dms krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:59:49.304430 4032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 647a4ef83d964adc9c6b66780d2be32f: No bootstrap required, opened a new log
I20260929 20:59:49.310932 4035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:49.290260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60288 (local address 127.0.97.62:39881)
0929 20:59:49.290978 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:49.290989 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:49.291026 (+ 37us) server_negotiation.cc:407] Connection header received
0929 20:59:49.292578 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:49.292600 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:49.293054 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:49.293363 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:49.295095 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:49.296301 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:49.297999 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:49.298634 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:49.298784 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:49.303637 (+ 4853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:49.303646 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:49.303661 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:49.303698 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:49.306915 (+ 3217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:49.307693 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:49.307699 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:49.307705 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:49.307880 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:49.308289 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:49.308293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:49.308295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:49.308814 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:49.309115 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:49.309515 (+ 400us) server_negotiation.cc:299] Negotiation successful
0929 20:59:49.309912 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":343,"threads_started":1}
I20260929 20:59:49.313474 4032 raft_consensus.cc:348] T 00000000000000000000000000000000 P 647a4ef83d964adc9c6b66780d2be32f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "647a4ef83d964adc9c6b66780d2be32f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39881 } }
I20260929 20:59:49.313908 4032 raft_consensus.cc:374] T 00000000000000000000000000000000 P 647a4ef83d964adc9c6b66780d2be32f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:49.314039 4032 raft_consensus.cc:729] T 00000000000000000000000000000000 P 647a4ef83d964adc9c6b66780d2be32f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 647a4ef83d964adc9c6b66780d2be32f, State: Initialized, Role: FOLLOWER
I20260929 20:59:49.314626 4032 consensus_queue.cc:260] T 00000000000000000000000000000000 P 647a4ef83d964adc9c6b66780d2be32f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "647a4ef83d964adc9c6b66780d2be32f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39881 } }
I20260929 20:59:49.314855 4032 raft_consensus.cc:388] T 00000000000000000000000000000000 P 647a4ef83d964adc9c6b66780d2be32f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:59:49.315001 4032 raft_consensus.cc:482] T 00000000000000000000000000000000 P 647a4ef83d964adc9c6b66780d2be32f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:59:49.315163 4032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 647a4ef83d964adc9c6b66780d2be32f [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:49.317523 4032 raft_consensus.cc:504] T 00000000000000000000000000000000 P 647a4ef83d964adc9c6b66780d2be32f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "647a4ef83d964adc9c6b66780d2be32f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39881 } }
I20260929 20:59:49.317986 4032 leader_election.cc:302] T 00000000000000000000000000000000 P 647a4ef83d964adc9c6b66780d2be32f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 647a4ef83d964adc9c6b66780d2be32f; no voters:
I20260929 20:59:49.318528 4032 leader_election.cc:288] T 00000000000000000000000000000000 P 647a4ef83d964adc9c6b66780d2be32f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:59:49.318665 4037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 647a4ef83d964adc9c6b66780d2be32f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:59:49.319348 4037 raft_consensus.cc:686] T 00000000000000000000000000000000 P 647a4ef83d964adc9c6b66780d2be32f [term 1 LEADER]: Becoming Leader. State: Replica: 647a4ef83d964adc9c6b66780d2be32f, State: Running, Role: LEADER
I20260929 20:59:49.319869 4037 consensus_queue.cc:237] T 00000000000000000000000000000000 P 647a4ef83d964adc9c6b66780d2be32f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "647a4ef83d964adc9c6b66780d2be32f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39881 } }
I20260929 20:59:49.320500 4032 sys_catalog.cc:564] T 00000000000000000000000000000000 P 647a4ef83d964adc9c6b66780d2be32f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:59:49.323436 4038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 647a4ef83d964adc9c6b66780d2be32f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "647a4ef83d964adc9c6b66780d2be32f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "647a4ef83d964adc9c6b66780d2be32f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39881 } } }
I20260929 20:59:49.323945 4038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 647a4ef83d964adc9c6b66780d2be32f [sys.catalog]: This master's current role is: LEADER
I20260929 20:59:49.323464 4039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 647a4ef83d964adc9c6b66780d2be32f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 647a4ef83d964adc9c6b66780d2be32f. Latest consensus state: current_term: 1 leader_uuid: "647a4ef83d964adc9c6b66780d2be32f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "647a4ef83d964adc9c6b66780d2be32f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39881 } } }
I20260929 20:59:49.324440 4039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 647a4ef83d964adc9c6b66780d2be32f [sys.catalog]: This master's current role is: LEADER
I20260929 20:59:49.325417 4045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:59:49.330283 4045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:59:49.338312 4045 catalog_manager.cc:1347] Generated new cluster ID: a97c9c35260745a78ded4b6bb655cc9b
I20260929 20:59:49.338441 4045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:59:49.363161 4045 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:59:49.364367 4045 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:59:49.374506 4045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 647a4ef83d964adc9c6b66780d2be32f: Generated new TSK 0
I20260929 20:59:49.375311 4045 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:59:49.425627 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39881
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:59:49.743912 4058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:49.744184 4058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:49.744333 4058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:49.751541 4058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:49.751643 4058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:49.751806 4058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:59:49.761319 4058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:49.762581 4058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:49.763881 4058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:49.777004 4066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:49.777140 4063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:49.777194 4064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:49.778000 4058 server_base.cc:1034] running on GCE node
I20260929 20:59:49.778523 4058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:49.779321 4058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:49.780503 4058 hybrid_clock.cc:648] HybridClock initialized: now 1790715589780439 us; error 53 us; skew 500 ppm
I20260929 20:59:49.780793 4058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:59:49 dist-test-slave-1dms krb5kdc[3952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:59:49.783738 4058 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:59:49.785184 4058 webserver.cc:460] Webserver started at http://127.0.97.1:35745/ using document root <none> and password file <none>
I20260929 20:59:49.785840 4058 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:49.785964 4058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:49.786283 4058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:49.789499 4058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-0/data/instance:
uuid: "c057a1344ed44e39800cc31c1df273e3"
format_stamp: "Formatted at 2026-09-29 20:59:49 on dist-test-slave-1dms"
server_key: "a2fbeeea1549abd7d4b9c6cee426fe2a"
server_key_iv: "bb656923aa9c56b3539a66ccf81385c8"
server_key_version: "encryptionkey@0"
I20260929 20:59:49.790268 4058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c057a1344ed44e39800cc31c1df273e3"
format_stamp: "Formatted at 2026-09-29 20:59:49 on dist-test-slave-1dms"
server_key: "a2fbeeea1549abd7d4b9c6cee426fe2a"
server_key_iv: "bb656923aa9c56b3539a66ccf81385c8"
server_key_version: "encryptionkey@0"
I20260929 20:59:49.795068 4058 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 20:59:49.798249 4073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:49.799513 4058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:59:49.799676 4058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-0/wal
uuid: "c057a1344ed44e39800cc31c1df273e3"
format_stamp: "Formatted at 2026-09-29 20:59:49 on dist-test-slave-1dms"
server_key: "a2fbeeea1549abd7d4b9c6cee426fe2a"
server_key_iv: "bb656923aa9c56b3539a66ccf81385c8"
server_key_version: "encryptionkey@0"
I20260929 20:59:49.799872 4058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:49.830511 4058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:49.834393 4058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:49.834669 4058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:49.835574 4058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:59:49.836815 4058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:59:49.836962 4058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:49.837100 4058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:59:49.837176 4058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:49.869207 4058 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39345
I20260929 20:59:49.869258 4185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39345 every 8 connection(s)
I20260929 20:59:49.870483 4058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:59:49.875138 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 4058
I20260929 20:59:49.875361 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:59:49.875936 388 external_mini_cluster.cc:1428] Setting key 88d1c4c03f6381fdfe93ece4ce0cd400
Sep 29 20:59:49 dist-test-slave-1dms krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:59:49.890574 4188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:49.873011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44545 (local address 127.0.97.62:39881)
0929 20:59:49.873489 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:49.873501 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:49.874652 (+ 1151us) server_negotiation.cc:407] Connection header received
0929 20:59:49.876686 (+ 2034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:49.876692 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:49.876779 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:49.876944 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:49.878388 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:49.879123 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:49.880769 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:49.881234 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:49.881376 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:49.884931 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:49.884936 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:49.884939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:49.884969 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:49.887650 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:49.888347 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:49.888354 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:49.888357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:49.888474 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:49.888936 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:49.888941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:49.888944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:49.889202 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:49.889358 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:49.890017 (+ 659us) server_negotiation.cc:299] Negotiation successful
0929 20:59:49.890232 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":219,"threads_started":1}
I20260929 20:59:49.891541 4189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:49.873599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39881 (local address 127.0.97.1:44545)
0929 20:59:49.874446 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:49.874504 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:49.876362 (+ 1858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:49.877177 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:49.877186 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:49.877684 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:49.878225 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:49.878241 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:49.879279 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:49.879283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:49.880513 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:49.880525 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:49.881461 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:49.881471 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:49.881708 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:49.882377 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:59:49.882398 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:49.884690 (+ 2292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:49.887789 (+ 3099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:49.887798 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:49.887814 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:49.888226 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:49.888590 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:49.888595 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:49.888599 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:49.888781 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:49.889334 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:49.889342 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:49.889662 (+ 320us) client_negotiation.cc:769] Sending connection context
0929 20:59:49.889928 (+ 266us) client_negotiation.cc:240] Negotiation successful
0929 20:59:49.890433 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":408,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 20:59:49.893175 4186 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39881
I20260929 20:59:49.893453 4186 heartbeater.cc:464] Registering TS with master...
I20260929 20:59:49.894223 4186 heartbeater.cc:511] Master 127.0.97.62:39881 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 20:59:49.896764 3997 ts_manager.cc:194] Registered new tserver with Master: c057a1344ed44e39800cc31c1df273e3 (127.0.97.1:39345)
I20260929 20:59:49.898914 3997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44545
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:59:49.920277 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39881
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:59:50.236822 4192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:50.237155 4192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:50.237326 4192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:50.244328 4192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:50.244465 4192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:50.244648 4192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:59:50.254134 4192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:50.255390 4192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:50.256664 4192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:50.269068 4200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:50.269110 4197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:50.269147 4198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:50.269871 4192 server_base.cc:1034] running on GCE node
I20260929 20:59:50.270395 4192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:50.271200 4192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:50.272395 4192 hybrid_clock.cc:648] HybridClock initialized: now 1790715590272348 us; error 40 us; skew 500 ppm
I20260929 20:59:50.272651 4192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:59:50 dist-test-slave-1dms krb5kdc[3952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:59:50.275725 4192 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:59:50.277171 4192 webserver.cc:460] Webserver started at http://127.0.97.2:45387/ using document root <none> and password file <none>
I20260929 20:59:50.277778 4192 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:50.277873 4192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:50.278105 4192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:50.280983 4192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-1/data/instance:
uuid: "0b89a4756d84478eab5145516c0503be"
format_stamp: "Formatted at 2026-09-29 20:59:50 on dist-test-slave-1dms"
server_key: "5bc2c5332ecc5cbc21d6c604069ab950"
server_key_iv: "2919b17157d299c2247e1a0c44430ed6"
server_key_version: "encryptionkey@0"
I20260929 20:59:50.281724 4192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0b89a4756d84478eab5145516c0503be"
format_stamp: "Formatted at 2026-09-29 20:59:50 on dist-test-slave-1dms"
server_key: "5bc2c5332ecc5cbc21d6c604069ab950"
server_key_iv: "2919b17157d299c2247e1a0c44430ed6"
server_key_version: "encryptionkey@0"
I20260929 20:59:50.286597 4192 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:59:50.289935 4207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:50.291196 4192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:59:50.291356 4192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-1/wal
uuid: "0b89a4756d84478eab5145516c0503be"
format_stamp: "Formatted at 2026-09-29 20:59:50 on dist-test-slave-1dms"
server_key: "5bc2c5332ecc5cbc21d6c604069ab950"
server_key_iv: "2919b17157d299c2247e1a0c44430ed6"
server_key_version: "encryptionkey@0"
I20260929 20:59:50.291522 4192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:50.320374 4192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:50.324594 4192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:50.324941 4192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:50.326028 4192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:59:50.327543 4192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:59:50.327611 4192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:50.327749 4192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:59:50.327809 4192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:50.361754 4192 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36935
I20260929 20:59:50.361840 4319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36935 every 8 connection(s)
I20260929 20:59:50.363003 4192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:59:50.367638 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 4192
I20260929 20:59:50.367841 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:59:50.368255 388 external_mini_cluster.cc:1428] Setting key 71e8ef1904e676960bfcec2e2cb0937a
Sep 29 20:59:50 dist-test-slave-1dms krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:59:50.383450 4188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:50.365458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37993 (local address 127.0.97.62:39881)
0929 20:59:50.365637 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:50.365642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:50.366818 (+ 1176us) server_negotiation.cc:407] Connection header received
0929 20:59:50.368622 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:50.368628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:50.368719 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:50.368873 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:50.370475 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:50.371633 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:50.373325 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:50.374070 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:50.374204 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:50.377861 (+ 3657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:50.377866 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:50.377869 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:50.377900 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:50.380359 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:50.381263 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:50.381266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:50.381269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:50.381383 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:50.381851 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:50.381854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:50.381856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:50.382086 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:50.382197 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:50.383017 (+ 820us) server_negotiation.cc:299] Negotiation successful
0929 20:59:50.383169 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 20:59:50.385085 4322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:50.365977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39881 (local address 127.0.97.2:37993)
0929 20:59:50.366640 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:50.366698 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:50.368327 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:50.369148 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:50.369156 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:50.369638 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:50.370313 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:50.370332 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:50.371814 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:50.371822 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:50.373178 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:50.373189 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:50.374273 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:50.374278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:50.374459 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:50.375260 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:59:50.375282 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:50.377661 (+ 2379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:50.380493 (+ 2832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:50.380504 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:50.380524 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:50.381150 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:50.381502 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:50.381510 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:50.381517 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:50.381739 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:50.382220 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:50.382232 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:50.382664 (+ 432us) client_negotiation.cc:769] Sending connection context
0929 20:59:50.382950 (+ 286us) client_negotiation.cc:240] Negotiation successful
0929 20:59:50.383508 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":273,"threads_started":1}
I20260929 20:59:50.386468 4320 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39881
I20260929 20:59:50.386766 4320 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 20:59:50.387428 4320 heartbeater.cc:511] Master 127.0.97.62:39881 requested a full tablet report, sending...
I20260929 20:59:50.388794 3997 ts_manager.cc:194] Registered new tserver with Master: 0b89a4756d84478eab5145516c0503be (127.0.97.2:36935)
I20260929 20:59:50.389582 3997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37993
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:59:50.412607 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39881
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:59:50.724319 4325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:50.724624 4325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:50.724790 4325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:50.731956 4325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:50.732091 4325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:50.732271 4325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:59:50.741367 4325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:50.742548 4325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:50.743850 4325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:50.756922 4330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:50.756985 4333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:50.756932 4331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:50.757773 4325 server_base.cc:1034] running on GCE node
I20260929 20:59:50.758322 4325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:50.759179 4325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:50.760392 4325 hybrid_clock.cc:648] HybridClock initialized: now 1790715590760360 us; error 45 us; skew 500 ppm
I20260929 20:59:50.760684 4325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:59:50 dist-test-slave-1dms krb5kdc[3952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:59:50.763693 4325 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:59:50.765275 4325 webserver.cc:460] Webserver started at http://127.0.97.3:44937/ using document root <none> and password file <none>
I20260929 20:59:50.765938 4325 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:50.766083 4325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:50.766378 4325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:50.769145 4325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-2/data/instance:
uuid: "6f5e925df7db461a9e6b2dc94d475ba3"
format_stamp: "Formatted at 2026-09-29 20:59:50 on dist-test-slave-1dms"
server_key: "7debbd4499bddede40dc95456009fd71"
server_key_iv: "6b50db210d66484f1ad72293c6564df8"
server_key_version: "encryptionkey@0"
I20260929 20:59:50.769876 4325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6f5e925df7db461a9e6b2dc94d475ba3"
format_stamp: "Formatted at 2026-09-29 20:59:50 on dist-test-slave-1dms"
server_key: "7debbd4499bddede40dc95456009fd71"
server_key_iv: "6b50db210d66484f1ad72293c6564df8"
server_key_version: "encryptionkey@0"
I20260929 20:59:50.774556 4325 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 20:59:50.777846 4340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:50.778976 4325 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 20:59:50.779141 4325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-2/wal
uuid: "6f5e925df7db461a9e6b2dc94d475ba3"
format_stamp: "Formatted at 2026-09-29 20:59:50 on dist-test-slave-1dms"
server_key: "7debbd4499bddede40dc95456009fd71"
server_key_iv: "6b50db210d66484f1ad72293c6564df8"
server_key_version: "encryptionkey@0"
I20260929 20:59:50.779336 4325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:50.791065 4325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:50.795030 4325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:50.795295 4325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:50.796221 4325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:59:50.797447 4325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:59:50.797547 4325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:50.797684 4325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:59:50.797761 4325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:50.829500 4325 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36725
I20260929 20:59:50.829577 4452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36725 every 8 connection(s)
I20260929 20:59:50.830677 4325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:59:50.838861 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 4325
I20260929 20:59:50.839048 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790715556572573-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:59:50.839448 388 external_mini_cluster.cc:1428] Setting key 57c1976eb397f4f46af6bf6f4a23d75b
Sep 29 20:59:50 dist-test-slave-1dms krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:59:50.849558 4188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:50.833029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35955 (local address 127.0.97.62:39881)
0929 20:59:50.833235 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:50.833241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:50.834369 (+ 1128us) server_negotiation.cc:407] Connection header received
0929 20:59:50.836044 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:50.836050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:50.836139 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:50.836250 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:50.837710 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:50.838452 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:50.840057 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:50.840547 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:50.840692 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:50.844109 (+ 3417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:50.844114 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:50.844117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:50.844154 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:50.846623 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:50.847343 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:50.847349 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:50.847352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:50.847494 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:50.847934 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:50.847939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:50.847942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:50.848202 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:50.848372 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:50.849016 (+ 644us) server_negotiation.cc:299] Negotiation successful
0929 20:59:50.849225 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:59:50.850358 4455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:50.833549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39881 (local address 127.0.97.3:35955)
0929 20:59:50.834214 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:50.834256 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:50.835773 (+ 1517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:50.836456 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:50.836464 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:50.837027 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:50.837564 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:50.837581 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:50.838600 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:50.838604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:50.839921 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:50.839932 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:50.840804 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:50.840812 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:50.841118 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:50.841794 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:59:50.841824 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:50.843940 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:50.846769 (+ 2829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:50.846782 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:50.846793 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:50.847230 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:50.847638 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:50.847644 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:50.847647 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:50.847817 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:50.848343 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:50.848350 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:50.848647 (+ 297us) client_negotiation.cc:769] Sending connection context
0929 20:59:50.848909 (+ 262us) client_negotiation.cc:240] Negotiation successful
0929 20:59:50.849372 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":270,"threads_started":1}
I20260929 20:59:50.851578 4453 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39881
I20260929 20:59:50.851826 4453 heartbeater.cc:464] Registering TS with master...
I20260929 20:59:50.852423 4453 heartbeater.cc:511] Master 127.0.97.62:39881 requested a full tablet report, sending...
I20260929 20:59:50.853811 3997 ts_manager.cc:194] Registered new tserver with Master: 6f5e925df7db461a9e6b2dc94d475ba3 (127.0.97.3:36725)
I20260929 20:59:50.854712 3997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:35955
I20260929 20:59:50.854733 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 20:59:50 dist-test-slave-1dms krb5kdc[3952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 20:59:50 dist-test-slave-1dms krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:59:50.901860 4460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:50.887513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37272 (local address 127.0.97.1:39345)
0929 20:59:50.888136 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:50.888146 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:50.888262 (+ 116us) server_negotiation.cc:407] Connection header received
0929 20:59:50.888481 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:50.888488 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:50.888721 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:50.888905 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:50.890269 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:50.891250 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:50.892467 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:50.893077 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:50.893206 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:50.896981 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:50.896991 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:50.897003 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:50.897032 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:50.899575 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:50.900142 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:50.900148 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:50.900150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:50.900257 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:50.900550 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:50.900554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:50.900556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:50.900945 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:50.901163 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:50.901392 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 20:59:50.901572 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":320,"threads_started":1}
I20260929 20:59:50.902274 4186 heartbeater.cc:503] Master 127.0.97.62:39881 was elected leader, sending a full tablet report...
Sep 29 20:59:50 dist-test-slave-1dms krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:59:50.918337 4188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:50.906129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60292 (local address 127.0.97.62:39881)
0929 20:59:50.906380 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:50.906389 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:50.906503 (+ 114us) server_negotiation.cc:407] Connection header received
0929 20:59:50.906613 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:50.906617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:50.906670 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:50.906749 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:50.907778 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:50.908443 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:50.909673 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:50.910076 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:50.910166 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:50.913663 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:50.913668 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:50.913671 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:50.913698 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:50.916133 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:50.916642 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:50.916646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:50.916648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:50.916755 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:50.917105 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:50.917108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:50.917110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:50.917414 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:50.917581 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:50.917865 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 20:59:50.918017 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
W20260929 20:59:50.918996 3997 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60292
I20260929 20:59:50.919903 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 4058
I20260929 20:59:50.931690 4460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:50.920882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37274 (local address 127.0.97.1:39345)
0929 20:59:50.921076 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:50.921082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:50.921266 (+ 184us) server_negotiation.cc:407] Connection header received
0929 20:59:50.921429 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:50.921434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:50.921534 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:50.921639 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:50.922771 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:50.923428 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:50.924605 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:50.925072 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:50.925202 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:50.927444 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:50.927448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:50.927452 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:50.927480 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:50.929771 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:50.930302 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:50.930306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:50.930308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:50.930409 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:50.930714 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:50.930717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:50.930719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:50.930935 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:50.931116 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:50.931337 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 20:59:50.931480 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:59:50.932250 4181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:59:51.013309 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 4058
I20260929 20:59:51.029801 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 4192
Sep 29 20:59:51 dist-test-slave-1dms krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:59:51.045081 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:51.030599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52728 (local address 127.0.97.2:36935)
0929 20:59:51.031081 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:51.031088 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:51.031113 (+ 25us) server_negotiation.cc:407] Connection header received
0929 20:59:51.031171 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:51.031177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:51.031395 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:51.031565 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:51.032726 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:51.033747 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:51.034980 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:51.035523 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:51.035657 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:51.039609 (+ 3952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:51.039619 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:51.039632 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:51.039670 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:51.042317 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:51.042946 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:51.042953 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:51.042955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:51.043074 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:51.043434 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:51.043441 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:51.043445 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:51.043797 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:51.044090 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:51.044430 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 20:59:51.044692 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":236,"threads_started":1}
I20260929 20:59:51.046288 4315 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:59:51.126299 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 4192
I20260929 20:59:51.141722 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 4325
Sep 29 20:59:51 dist-test-slave-1dms krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:59:51.157953 4467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:51.142730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37512 (local address 127.0.97.3:36725)
0929 20:59:51.143334 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:51.143341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:51.143367 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:59:51.143422 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:51.143429 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:51.143640 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:51.143761 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:51.145042 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:51.146418 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:51.147735 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:51.148349 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:51.148485 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:51.152618 (+ 4133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:51.152628 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:51.152641 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:51.152669 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:51.155327 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:51.155967 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:51.155979 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:51.155981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:51.156087 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:51.156442 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:51.156449 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:51.156452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:51.156770 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:51.157135 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:51.157380 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 20:59:51.157625 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":284,"threads_started":1}
I20260929 20:59:51.159150 4448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:59:51.241892 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 4325
I20260929 20:59:51.257437 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 3965
I20260929 20:59:51.258512 4027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:59:51.359747 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 3965
2026-09-29T20:59:51Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3756 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:59:51 dist-test-slave-1dms krb5kdc[4472](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:59:51 dist-test-slave-1dms krb5kdc[4472](info): set up 2 sockets
Sep 29 20:59:51 dist-test-slave-1dms krb5kdc[4472](info): commencing operation
krb5kdc: starting...
W20260929 20:59:52.457377 388 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 20:59:52 dist-test-slave-1dms krb5kdc[4472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:59:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:59:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:59:52.662575 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46399
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:59:52.994701 4485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:52.994985 4485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:52.995153 4485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:53.002285 4485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:59:53.002384 4485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:53.002478 4485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:53.002578 4485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:59:53.002630 4485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:59:53.012403 4485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:53.013705 4485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:53.015039 4485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:53.026774 4490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:53.026687 4491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:53.026734 4493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:53.027649 4485 server_base.cc:1034] running on GCE node
I20260929 20:59:53.028419 4485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:53.029793 4485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:53.030994 4485 hybrid_clock.cc:648] HybridClock initialized: now 1790715593030960 us; error 35 us; skew 500 ppm
I20260929 20:59:53.031347 4485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:59:53 dist-test-slave-1dms krb5kdc[4472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:59:53.034317 4485 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:59:53.035511 4485 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:53.035640 4485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:53.035953 4485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:53.038897 4485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "eb04e3d3d3734e84a870b8cff0c5cc3a"
format_stamp: "Formatted at 2026-09-29 20:59:53 on dist-test-slave-1dms"
server_key: "e60fdc46428ab5d2dcb8c9f75d404a3e"
server_key_iv: "7bd9a5b07def7c18139f70b407f684ac"
server_key_version: "encryptionkey@0"
I20260929 20:59:53.039681 4485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "eb04e3d3d3734e84a870b8cff0c5cc3a"
format_stamp: "Formatted at 2026-09-29 20:59:53 on dist-test-slave-1dms"
server_key: "e60fdc46428ab5d2dcb8c9f75d404a3e"
server_key_iv: "7bd9a5b07def7c18139f70b407f684ac"
server_key_version: "encryptionkey@0"
I20260929 20:59:53.044552 4485 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 20:59:53.048043 4499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:53.049337 4485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:59:53.049508 4485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "eb04e3d3d3734e84a870b8cff0c5cc3a"
format_stamp: "Formatted at 2026-09-29 20:59:53 on dist-test-slave-1dms"
server_key: "e60fdc46428ab5d2dcb8c9f75d404a3e"
server_key_iv: "7bd9a5b07def7c18139f70b407f684ac"
server_key_version: "encryptionkey@0"
I20260929 20:59:53.049722 4485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:53.077467 4485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:53.081776 4485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:53.082048 4485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:53.100296 4485 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46399
I20260929 20:59:53.100277 4550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46399 every 8 connection(s)
I20260929 20:59:53.101308 4485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:59:53.104872 4551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:53.111018 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 4485
I20260929 20:59:53.111249 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 20:59:53.111831 388 external_mini_cluster.cc:1428] Setting key cc25f66c68a09ff8f692e3dd776a6014
I20260929 20:59:53.115272 4551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb04e3d3d3734e84a870b8cff0c5cc3a: Bootstrap starting.
I20260929 20:59:53.118990 4551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb04e3d3d3734e84a870b8cff0c5cc3a: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:53.120075 4551 log.cc:824] T 00000000000000000000000000000000 P eb04e3d3d3734e84a870b8cff0c5cc3a: Log is configured to *not* fsync() on all Append() calls
I20260929 20:59:53.122776 4551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb04e3d3d3734e84a870b8cff0c5cc3a: No bootstrap required, opened a new log
Sep 29 20:59:53 dist-test-slave-1dms krb5kdc[4472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:59:53.131341 4551 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb04e3d3d3734e84a870b8cff0c5cc3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb04e3d3d3734e84a870b8cff0c5cc3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46399 } }
I20260929 20:59:53.131791 4551 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb04e3d3d3734e84a870b8cff0c5cc3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:53.131892 4551 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb04e3d3d3734e84a870b8cff0c5cc3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb04e3d3d3734e84a870b8cff0c5cc3a, State: Initialized, Role: FOLLOWER
I20260929 20:59:53.132565 4551 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb04e3d3d3734e84a870b8cff0c5cc3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb04e3d3d3734e84a870b8cff0c5cc3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46399 } }
I20260929 20:59:53.132810 4551 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb04e3d3d3734e84a870b8cff0c5cc3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:59:53.133024 4551 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb04e3d3d3734e84a870b8cff0c5cc3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:59:53.133257 4551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb04e3d3d3734e84a870b8cff0c5cc3a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:53.134392 4554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:53.113820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38134 (local address 127.0.97.62:46399)
0929 20:59:53.114576 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:53.114590 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:53.114637 (+ 47us) server_negotiation.cc:407] Connection header received
0929 20:59:53.116986 (+ 2349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:53.117020 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:53.117531 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:53.117931 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:53.119318 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:53.120740 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:53.122030 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:53.122681 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:53.122884 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:53.127052 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:53.127060 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:53.127074 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:53.127114 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:53.130189 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:53.130948 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:53.130954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:53.130960 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:53.131116 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:53.131519 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:53.131521 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:53.131523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:53.132154 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:53.132433 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:53.132913 (+ 480us) server_negotiation.cc:299] Negotiation successful
0929 20:59:53.133311 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":311,"threads_started":1}
I20260929 20:59:53.135627 4551 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb04e3d3d3734e84a870b8cff0c5cc3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb04e3d3d3734e84a870b8cff0c5cc3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46399 } }
I20260929 20:59:53.136117 4551 leader_election.cc:302] T 00000000000000000000000000000000 P eb04e3d3d3734e84a870b8cff0c5cc3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb04e3d3d3734e84a870b8cff0c5cc3a; no voters:
I20260929 20:59:53.136727 4551 leader_election.cc:288] T 00000000000000000000000000000000 P eb04e3d3d3734e84a870b8cff0c5cc3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:59:53.136900 4556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb04e3d3d3734e84a870b8cff0c5cc3a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:59:53.138139 4556 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb04e3d3d3734e84a870b8cff0c5cc3a [term 1 LEADER]: Becoming Leader. State: Replica: eb04e3d3d3734e84a870b8cff0c5cc3a, State: Running, Role: LEADER
I20260929 20:59:53.138774 4556 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb04e3d3d3734e84a870b8cff0c5cc3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb04e3d3d3734e84a870b8cff0c5cc3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46399 } }
I20260929 20:59:53.139045 4551 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb04e3d3d3734e84a870b8cff0c5cc3a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:59:53.141695 4558 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb04e3d3d3734e84a870b8cff0c5cc3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb04e3d3d3734e84a870b8cff0c5cc3a. Latest consensus state: current_term: 1 leader_uuid: "eb04e3d3d3734e84a870b8cff0c5cc3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb04e3d3d3734e84a870b8cff0c5cc3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46399 } } }
I20260929 20:59:53.142102 4558 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb04e3d3d3734e84a870b8cff0c5cc3a [sys.catalog]: This master's current role is: LEADER
I20260929 20:59:53.142609 4557 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb04e3d3d3734e84a870b8cff0c5cc3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb04e3d3d3734e84a870b8cff0c5cc3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb04e3d3d3734e84a870b8cff0c5cc3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46399 } } }
I20260929 20:59:53.142977 4557 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb04e3d3d3734e84a870b8cff0c5cc3a [sys.catalog]: This master's current role is: LEADER
I20260929 20:59:53.143644 4565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:59:53.148896 4565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:59:53.157826 4565 catalog_manager.cc:1347] Generated new cluster ID: 4121bdf4e75a428ca9ac535c333d1818
I20260929 20:59:53.157958 4565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:59:53.175330 4565 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:59:53.176656 4565 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:59:53.183220 4565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb04e3d3d3734e84a870b8cff0c5cc3a: Generated new TSK 0
I20260929 20:59:53.184002 4565 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:59:53.238169 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46399
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:59:53.563592 4577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:53.563848 4577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:53.564013 4577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:53.571074 4577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:53.571198 4577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:53.571377 4577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:59:53.580655 4577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:53.581949 4577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:53.583205 4577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:53.595398 4582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:53.595435 4583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:53.595603 4585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:53.596371 4577 server_base.cc:1034] running on GCE node
I20260929 20:59:53.597005 4577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:53.597894 4577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:53.599119 4577 hybrid_clock.cc:648] HybridClock initialized: now 1790715593599078 us; error 50 us; skew 500 ppm
I20260929 20:59:53.599431 4577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:59:53 dist-test-slave-1dms krb5kdc[4472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:59:53.602634 4577 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:59:53.603835 4577 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:53.603965 4577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:53.604277 4577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:53.607179 4577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-0/data/instance:
uuid: "49f61222e1564771bb7a79efabec672b"
format_stamp: "Formatted at 2026-09-29 20:59:53 on dist-test-slave-1dms"
server_key: "845536adf98aad38a9b722eefe1a4b46"
server_key_iv: "365b00969e5d836515f47074d81117e3"
server_key_version: "encryptionkey@0"
I20260929 20:59:53.607959 4577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-0/wal/instance:
uuid: "49f61222e1564771bb7a79efabec672b"
format_stamp: "Formatted at 2026-09-29 20:59:53 on dist-test-slave-1dms"
server_key: "845536adf98aad38a9b722eefe1a4b46"
server_key_iv: "365b00969e5d836515f47074d81117e3"
server_key_version: "encryptionkey@0"
I20260929 20:59:53.612591 4577 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260929 20:59:53.615865 4591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:53.617233 4577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:59:53.617400 4577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-0/wal
uuid: "49f61222e1564771bb7a79efabec672b"
format_stamp: "Formatted at 2026-09-29 20:59:53 on dist-test-slave-1dms"
server_key: "845536adf98aad38a9b722eefe1a4b46"
server_key_iv: "365b00969e5d836515f47074d81117e3"
server_key_version: "encryptionkey@0"
I20260929 20:59:53.617611 4577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:53.649204 4577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:53.653738 4577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:53.654062 4577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:53.654986 4577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:59:53.656255 4577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:59:53.656376 4577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:53.656503 4577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:59:53.656589 4577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:53.688585 4577 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37871
I20260929 20:59:53.688634 4703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37871 every 8 connection(s)
I20260929 20:59:53.689465 4577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:59:53.698060 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 4577
I20260929 20:59:53.698271 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:59:53.698854 388 external_mini_cluster.cc:1428] Setting key ae7f1c87d3a08712839d08c4d430616c
Sep 29 20:59:53 dist-test-slave-1dms krb5kdc[4472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:59:53.708124 4706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:53.691813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54071 (local address 127.0.97.62:46399)
0929 20:59:53.692246 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:53.692250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:53.693158 (+ 908us) server_negotiation.cc:407] Connection header received
0929 20:59:53.694893 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:53.694899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:53.694994 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:53.695113 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:53.696578 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:53.697340 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:53.698988 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:53.699387 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:53.699523 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:53.702826 (+ 3303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:53.702830 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:53.702834 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:53.702862 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:53.705282 (+ 2420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:53.705978 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:53.705986 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:53.705989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:53.706104 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:53.706516 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:53.706521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:53.706525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:53.706769 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:53.706918 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:53.707568 (+ 650us) server_negotiation.cc:299] Negotiation successful
0929 20:59:53.707777 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":236,"threads_started":1}
I20260929 20:59:53.709097 4707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:53.692276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46399 (local address 127.0.97.1:54071)
0929 20:59:53.693015 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:53.693052 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:53.694630 (+ 1578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:53.695331 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:53.695359 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:53.695894 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:53.696415 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:53.696432 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:53.697505 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:53.697510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:53.698842 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:53.698854 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:53.699602 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:53.699611 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:53.699853 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:53.700503 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:59:53.700524 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:53.702611 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:53.705413 (+ 2802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:53.705421 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:53.705437 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:53.705861 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:53.706220 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:53.706226 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:53.706230 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:53.706405 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:53.706893 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:53.706900 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:53.707213 (+ 313us) client_negotiation.cc:769] Sending connection context
0929 20:59:53.707469 (+ 256us) client_negotiation.cc:240] Negotiation successful
0929 20:59:53.707962 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":272,"threads_started":1}
I20260929 20:59:53.710587 4704 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46399
I20260929 20:59:53.710906 4704 heartbeater.cc:464] Registering TS with master...
I20260929 20:59:53.711640 4704 heartbeater.cc:511] Master 127.0.97.62:46399 requested a full tablet report, sending...
I20260929 20:59:53.713716 4516 ts_manager.cc:194] Registered new tserver with Master: 49f61222e1564771bb7a79efabec672b (127.0.97.1:37871)
I20260929 20:59:53.715091 4516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54071
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:59:53.741519 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46399
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:59:54.054901 4710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:54.055174 4710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:54.055338 4710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:54.062512 4710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:54.062650 4710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:54.062840 4710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:59:54.072461 4710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:54.073796 4710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:54.075129 4710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:54.087867 4715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:54.087894 4718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:54.087845 4716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:54.088609 4710 server_base.cc:1034] running on GCE node
I20260929 20:59:54.089332 4710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:54.090260 4710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:54.091490 4710 hybrid_clock.cc:648] HybridClock initialized: now 1790715594091452 us; error 65 us; skew 500 ppm
I20260929 20:59:54.091813 4710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:59:54 dist-test-slave-1dms krb5kdc[4472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:59:54.094923 4710 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:59:54.096078 4710 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:54.096225 4710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:54.096627 4710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:54.099861 4710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-1/data/instance:
uuid: "7827899f0e3240b0967165ac05e56cde"
format_stamp: "Formatted at 2026-09-29 20:59:54 on dist-test-slave-1dms"
server_key: "a288d25be7ac40e673a6673e542a4f5a"
server_key_iv: "dbafd137cff15a61e5624331c147c627"
server_key_version: "encryptionkey@0"
I20260929 20:59:54.100679 4710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7827899f0e3240b0967165ac05e56cde"
format_stamp: "Formatted at 2026-09-29 20:59:54 on dist-test-slave-1dms"
server_key: "a288d25be7ac40e673a6673e542a4f5a"
server_key_iv: "dbafd137cff15a61e5624331c147c627"
server_key_version: "encryptionkey@0"
I20260929 20:59:54.105648 4710 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260929 20:59:54.109056 4724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:54.110428 4710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:59:54.110591 4710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-1/wal
uuid: "7827899f0e3240b0967165ac05e56cde"
format_stamp: "Formatted at 2026-09-29 20:59:54 on dist-test-slave-1dms"
server_key: "a288d25be7ac40e673a6673e542a4f5a"
server_key_iv: "dbafd137cff15a61e5624331c147c627"
server_key_version: "encryptionkey@0"
I20260929 20:59:54.110805 4710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:54.125607 4710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:54.129977 4710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:54.130255 4710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:54.131219 4710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:59:54.132513 4710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:59:54.132597 4710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:54.132754 4710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:59:54.132820 4710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:54.164400 4710 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35337
I20260929 20:59:54.164443 4836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35337 every 8 connection(s)
I20260929 20:59:54.165302 4710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:59:54.168228 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 4710
I20260929 20:59:54.168447 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:59:54.169034 388 external_mini_cluster.cc:1428] Setting key 88a2f871cd866acc598c4d147e006570
Sep 29 20:59:54 dist-test-slave-1dms krb5kdc[4472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:59:54.187781 4706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:54.167742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45709 (local address 127.0.97.62:46399)
0929 20:59:54.167953 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:54.167959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:54.169133 (+ 1174us) server_negotiation.cc:407] Connection header received
0929 20:59:54.171474 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:54.171478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:54.171537 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:54.171608 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:54.173724 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:54.174418 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:54.176894 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:54.177324 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:54.177419 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:54.180920 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:54.180928 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:54.180935 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:54.180976 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:54.184561 (+ 3585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:54.185400 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:54.185407 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:54.185413 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:54.185558 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:54.185970 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:54.185977 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:54.185983 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:54.186383 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:54.186549 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:54.187168 (+ 619us) server_negotiation.cc:299] Negotiation successful
0929 20:59:54.187411 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:59:54.188521 4839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:54.168120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46399 (local address 127.0.97.2:45709)
0929 20:59:54.168958 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:54.169006 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:54.171167 (+ 2161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:54.171899 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:54.171913 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:54.172701 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:54.173541 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:54.173567 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:54.174567 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:54.174575 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:54.176706 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:54.176721 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:54.177558 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:54.177566 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:54.177879 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:54.178639 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:59:54.178664 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:54.180674 (+ 2010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:54.184707 (+ 4033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:54.184713 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:54.184725 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:54.185285 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:54.185677 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:54.185680 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:54.185683 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:54.185847 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:54.186506 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:54.186511 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:54.186807 (+ 296us) client_negotiation.cc:769] Sending connection context
0929 20:59:54.187020 (+ 213us) client_negotiation.cc:240] Negotiation successful
0929 20:59:54.187375 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":345,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 20:59:54.190033 4837 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46399
I20260929 20:59:54.190294 4837 heartbeater.cc:464] Registering TS with master...
I20260929 20:59:54.191015 4837 heartbeater.cc:511] Master 127.0.97.62:46399 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 20:59:54.192378 4516 ts_manager.cc:194] Registered new tserver with Master: 7827899f0e3240b0967165ac05e56cde (127.0.97.2:35337)
I20260929 20:59:54.193641 4516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45709
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:59:54.216320 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46399
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:59:54.531560 4842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:54.531834 4842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:54.532012 4842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:54.539135 4842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:54.539261 4842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:54.539449 4842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:59:54.549052 4842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:54.550305 4842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:54.551568 4842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:54.564698 4847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:54.564738 4848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:54.564671 4850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:54.565297 4842 server_base.cc:1034] running on GCE node
I20260929 20:59:54.565905 4842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:54.566807 4842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:54.568028 4842 hybrid_clock.cc:648] HybridClock initialized: now 1790715594567980 us; error 62 us; skew 500 ppm
I20260929 20:59:54.568341 4842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:59:54 dist-test-slave-1dms krb5kdc[4472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:59:54.571316 4842 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:59:54.572410 4842 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:54.572540 4842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:54.572916 4842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:54.575664 4842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-2/data/instance:
uuid: "e49c0e2d569b43c9b70ab030d22e63dc"
format_stamp: "Formatted at 2026-09-29 20:59:54 on dist-test-slave-1dms"
server_key: "be2c3087fcf173070bd7b1ea7165814d"
server_key_iv: "995da5478d2ef6c0c49a2ef837f49cad"
server_key_version: "encryptionkey@0"
I20260929 20:59:54.576457 4842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e49c0e2d569b43c9b70ab030d22e63dc"
format_stamp: "Formatted at 2026-09-29 20:59:54 on dist-test-slave-1dms"
server_key: "be2c3087fcf173070bd7b1ea7165814d"
server_key_iv: "995da5478d2ef6c0c49a2ef837f49cad"
server_key_version: "encryptionkey@0"
I20260929 20:59:54.581270 4842 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 20:59:54.584581 4856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:54.585879 4842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:59:54.586050 4842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-2/wal
uuid: "e49c0e2d569b43c9b70ab030d22e63dc"
format_stamp: "Formatted at 2026-09-29 20:59:54 on dist-test-slave-1dms"
server_key: "be2c3087fcf173070bd7b1ea7165814d"
server_key_iv: "995da5478d2ef6c0c49a2ef837f49cad"
server_key_version: "encryptionkey@0"
I20260929 20:59:54.586257 4842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:54.605805 4842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:54.609550 4842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:54.609823 4842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:54.610704 4842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:59:54.611931 4842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:59:54.612051 4842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:54.612183 4842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:59:54.612274 4842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:54.644917 4842 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40777
I20260929 20:59:54.644963 4968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40777 every 8 connection(s)
I20260929 20:59:54.645792 4842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:59:54.653410 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 4842
I20260929 20:59:54.653599 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:59:54.653991 388 external_mini_cluster.cc:1428] Setting key 94061aadd6db592d21fd9bc05b4fab67
Sep 29 20:59:54 dist-test-slave-1dms krb5kdc[4472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:59:54.664486 4706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:54.648113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45031 (local address 127.0.97.62:46399)
0929 20:59:54.648289 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:54.648294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:54.649458 (+ 1164us) server_negotiation.cc:407] Connection header received
0929 20:59:54.651183 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:54.651189 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:54.651284 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:54.651395 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:54.652797 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:54.653900 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:54.655568 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:54.655981 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:54.656115 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:54.659342 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:54.659350 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:54.659359 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:54.659388 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:54.661661 (+ 2273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:54.662330 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:54.662337 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:54.662341 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:54.662457 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:54.662864 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:54.662870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:54.662873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:54.663127 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:54.663300 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:54.663934 (+ 634us) server_negotiation.cc:299] Negotiation successful
0929 20:59:54.664134 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 20:59:54.665395 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:54.648594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46399 (local address 127.0.97.3:45031)
0929 20:59:54.649312 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:54.649349 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:54.650933 (+ 1584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:54.651607 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:54.651616 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:54.652106 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:54.652624 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:54.652640 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:54.654037 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:54.654049 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:54.655410 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:54.655421 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:54.656195 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:54.656204 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:54.656438 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:54.657134 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:59:54.657155 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:54.659120 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:54.661797 (+ 2677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:54.661806 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:54.661820 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:54.662213 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:54.662566 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:59:54.662571 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:59:54.662575 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:59:54.662753 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:59:54.663282 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:54.663289 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:54.663578 (+ 289us) client_negotiation.cc:769] Sending connection context
0929 20:59:54.663835 (+ 257us) client_negotiation.cc:240] Negotiation successful
0929 20:59:54.664301 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":328,"threads_started":1}
I20260929 20:59:54.666702 4969 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46399
I20260929 20:59:54.667016 4969 heartbeater.cc:464] Registering TS with master...
I20260929 20:59:54.667629 4969 heartbeater.cc:511] Master 127.0.97.62:46399 requested a full tablet report, sending...
I20260929 20:59:54.669049 4516 ts_manager.cc:194] Registered new tserver with Master: e49c0e2d569b43c9b70ab030d22e63dc (127.0.97.3:40777)
I20260929 20:59:54.669813 4516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45031
I20260929 20:59:54.680644 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:59:54.697760 4706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:54.685160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38150 (local address 127.0.97.62:46399)
0929 20:59:54.685345 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:54.685349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:54.685847 (+ 498us) server_negotiation.cc:407] Connection header received
0929 20:59:54.685974 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:54.685978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:54.686037 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:54.686137 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:54.687525 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:54.688234 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:54.689627 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:54.690296 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:54.690447 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:54.692816 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:54.692822 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:54.692827 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:54.692889 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:54.695570 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:54.696160 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:54.696164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:54.696166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:54.696277 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:54.696557 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:54.696560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:54.696562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:54.696826 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:54.697053 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:54.697313 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 20:59:54.697512 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:59:54.701514 4516 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:59:54.708094 4516 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:59:54.718236 4704 heartbeater.cc:503] Master 127.0.97.62:46399 was elected leader, sending a full tablet report...
I20260929 20:59:54.729828 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:54.720037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40777 (local address 127.0.0.1:48704)
0929 20:59:54.721028 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:54.721057 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:54.721244 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:54.722295 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:54.722306 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:54.722351 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:59:54.722656 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:54.722669 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:54.724579 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:54.724584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:54.726920 (+ 2336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:54.726935 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:54.728855 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:54.728863 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:54.729063 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:54.729129 (+ 66us) client_negotiation.cc:769] Sending connection context
0929 20:59:54.729261 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 20:59:54.729448 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"thread_start_us":647,"threads_started":1}
I20260929 20:59:54.729897 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:54.720255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48704 (local address 127.0.97.3:40777)
0929 20:59:54.720925 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:54.720934 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:54.721154 (+ 220us) server_negotiation.cc:407] Connection header received
0929 20:59:54.721447 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:54.721458 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:54.721704 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:54.721954 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:59:54.722868 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:54.724439 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:54.727134 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:54.728604 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:54.728737 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:54.728980 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:54.729425 (+ 445us) server_negotiation.cc:299] Negotiation successful
0929 20:59:54.729584 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":303,"threads_started":1}
I20260929 20:59:54.729982 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:54.721400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37871 (local address 127.0.0.1:55934)
0929 20:59:54.721991 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:54.722008 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:54.722118 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:54.723261 (+ 1143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:54.723268 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:54.723300 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:59:54.723835 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:54.723850 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:54.726071 (+ 2221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:54.726076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:54.727801 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:54.727811 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:54.729197 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:54.729213 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:54.729382 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:54.729414 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 20:59:54.729495 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 20:59:54.729624 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":263,"threads_started":1}
I20260929 20:59:54.731803 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:54.721997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35337 (local address 127.0.0.1:44706)
0929 20:59:54.723957 (+ 1960us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:54.723974 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:54.724071 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:54.724978 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:54.724982 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:54.725006 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:59:54.725377 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:54.725386 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:54.727111 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:54.727117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:54.729258 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:54.729269 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:54.731274 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:54.731282 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:54.731396 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:54.731418 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 20:59:54.731490 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 20:59:54.731563 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1828,"mutex_wait_us":1,"thread_start_us":1688,"threads_started":1}
I20260929 20:59:54.732023 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:54.722478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44706 (local address 127.0.97.2:35337)
0929 20:59:54.724485 (+ 2007us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:54.724495 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:54.724522 (+ 27us) server_negotiation.cc:407] Connection header received
0929 20:59:54.724606 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:54.724613 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:54.724773 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:54.724984 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:59:54.725524 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:54.726914 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:54.729423 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:54.731078 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:54.731213 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:54.731392 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:54.731556 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 20:59:54.731716 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1869,"thread_start_us":1786,"threads_started":1}
I20260929 20:59:54.732903 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:54.721568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55934 (local address 127.0.97.1:37871)
0929 20:59:54.722628 (+ 1060us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:54.722638 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:54.722669 (+ 31us) server_negotiation.cc:407] Connection header received
0929 20:59:54.722775 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:54.722785 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:54.723039 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:54.724094 (+ 1055us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:59:54.724333 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:54.725934 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:54.727945 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:54.728984 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:54.729105 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:54.729328 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:54.730729 (+ 1401us) server_negotiation.cc:299] Negotiation successful
0929 20:59:54.730891 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":908,"thread_start_us":756,"threads_started":1}
I20260929 20:59:54.734000 4904 tablet_service.cc:1467] Processing CreateTablet for tablet a2898dd00b974b40912e1f5c1f3f12ad (DEFAULT_TABLE table=test-table [id=0ad6737b25444202a5b78e4234e521d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:59:54.735522 4904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2898dd00b974b40912e1f5c1f3f12ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:54.736339 4772 tablet_service.cc:1467] Processing CreateTablet for tablet a2898dd00b974b40912e1f5c1f3f12ad (DEFAULT_TABLE table=test-table [id=0ad6737b25444202a5b78e4234e521d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:59:54.736899 4639 tablet_service.cc:1467] Processing CreateTablet for tablet a2898dd00b974b40912e1f5c1f3f12ad (DEFAULT_TABLE table=test-table [id=0ad6737b25444202a5b78e4234e521d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:59:54.737668 4772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2898dd00b974b40912e1f5c1f3f12ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:54.737802 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2898dd00b974b40912e1f5c1f3f12ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:54.749594 4987 tablet_bootstrap.cc:492] T a2898dd00b974b40912e1f5c1f3f12ad P 49f61222e1564771bb7a79efabec672b: Bootstrap starting.
I20260929 20:59:54.751523 4988 tablet_bootstrap.cc:492] T a2898dd00b974b40912e1f5c1f3f12ad P e49c0e2d569b43c9b70ab030d22e63dc: Bootstrap starting.
I20260929 20:59:54.753273 4987 tablet_bootstrap.cc:654] T a2898dd00b974b40912e1f5c1f3f12ad P 49f61222e1564771bb7a79efabec672b: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:54.754549 4987 log.cc:824] T a2898dd00b974b40912e1f5c1f3f12ad P 49f61222e1564771bb7a79efabec672b: Log is configured to *not* fsync() on all Append() calls
I20260929 20:59:54.755030 4988 tablet_bootstrap.cc:654] T a2898dd00b974b40912e1f5c1f3f12ad P e49c0e2d569b43c9b70ab030d22e63dc: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:54.756291 4988 log.cc:824] T a2898dd00b974b40912e1f5c1f3f12ad P e49c0e2d569b43c9b70ab030d22e63dc: Log is configured to *not* fsync() on all Append() calls
I20260929 20:59:54.757141 4989 tablet_bootstrap.cc:492] T a2898dd00b974b40912e1f5c1f3f12ad P 7827899f0e3240b0967165ac05e56cde: Bootstrap starting.
I20260929 20:59:54.757833 4987 tablet_bootstrap.cc:492] T a2898dd00b974b40912e1f5c1f3f12ad P 49f61222e1564771bb7a79efabec672b: No bootstrap required, opened a new log
I20260929 20:59:54.758091 4987 ts_tablet_manager.cc:1397] T a2898dd00b974b40912e1f5c1f3f12ad P 49f61222e1564771bb7a79efabec672b: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 20:59:54.759135 4988 tablet_bootstrap.cc:492] T a2898dd00b974b40912e1f5c1f3f12ad P e49c0e2d569b43c9b70ab030d22e63dc: No bootstrap required, opened a new log
I20260929 20:59:54.759380 4988 ts_tablet_manager.cc:1397] T a2898dd00b974b40912e1f5c1f3f12ad P e49c0e2d569b43c9b70ab030d22e63dc: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260929 20:59:54.760298 4989 tablet_bootstrap.cc:654] T a2898dd00b974b40912e1f5c1f3f12ad P 7827899f0e3240b0967165ac05e56cde: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:54.761432 4989 log.cc:824] T a2898dd00b974b40912e1f5c1f3f12ad P 7827899f0e3240b0967165ac05e56cde: Log is configured to *not* fsync() on all Append() calls
I20260929 20:59:54.764200 4989 tablet_bootstrap.cc:492] T a2898dd00b974b40912e1f5c1f3f12ad P 7827899f0e3240b0967165ac05e56cde: No bootstrap required, opened a new log
I20260929 20:59:54.764446 4989 ts_tablet_manager.cc:1397] T a2898dd00b974b40912e1f5c1f3f12ad P 7827899f0e3240b0967165ac05e56cde: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260929 20:59:54.766637 4988 raft_consensus.cc:348] T a2898dd00b974b40912e1f5c1f3f12ad P e49c0e2d569b43c9b70ab030d22e63dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49c0e2d569b43c9b70ab030d22e63dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40777 } } peers { permanent_uuid: "49f61222e1564771bb7a79efabec672b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37871 } } peers { permanent_uuid: "7827899f0e3240b0967165ac05e56cde" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35337 } }
I20260929 20:59:54.766986 4988 raft_consensus.cc:374] T a2898dd00b974b40912e1f5c1f3f12ad P e49c0e2d569b43c9b70ab030d22e63dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:54.767112 4988 raft_consensus.cc:729] T a2898dd00b974b40912e1f5c1f3f12ad P e49c0e2d569b43c9b70ab030d22e63dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e49c0e2d569b43c9b70ab030d22e63dc, State: Initialized, Role: FOLLOWER
I20260929 20:59:54.767699 4988 consensus_queue.cc:260] T a2898dd00b974b40912e1f5c1f3f12ad P e49c0e2d569b43c9b70ab030d22e63dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49c0e2d569b43c9b70ab030d22e63dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40777 } } peers { permanent_uuid: "49f61222e1564771bb7a79efabec672b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37871 } } peers { permanent_uuid: "7827899f0e3240b0967165ac05e56cde" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35337 } }
I20260929 20:59:54.768008 4987 raft_consensus.cc:348] T a2898dd00b974b40912e1f5c1f3f12ad P 49f61222e1564771bb7a79efabec672b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49c0e2d569b43c9b70ab030d22e63dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40777 } } peers { permanent_uuid: "49f61222e1564771bb7a79efabec672b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37871 } } peers { permanent_uuid: "7827899f0e3240b0967165ac05e56cde" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35337 } }
I20260929 20:59:54.768411 4987 raft_consensus.cc:374] T a2898dd00b974b40912e1f5c1f3f12ad P 49f61222e1564771bb7a79efabec672b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:54.768514 4987 raft_consensus.cc:729] T a2898dd00b974b40912e1f5c1f3f12ad P 49f61222e1564771bb7a79efabec672b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49f61222e1564771bb7a79efabec672b, State: Initialized, Role: FOLLOWER
I20260929 20:59:54.769192 4969 heartbeater.cc:503] Master 127.0.97.62:46399 was elected leader, sending a full tablet report...
I20260929 20:59:54.769093 4987 consensus_queue.cc:260] T a2898dd00b974b40912e1f5c1f3f12ad P 49f61222e1564771bb7a79efabec672b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49c0e2d569b43c9b70ab030d22e63dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40777 } } peers { permanent_uuid: "49f61222e1564771bb7a79efabec672b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37871 } } peers { permanent_uuid: "7827899f0e3240b0967165ac05e56cde" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35337 } }
I20260929 20:59:54.769816 4988 ts_tablet_manager.cc:1428] T a2898dd00b974b40912e1f5c1f3f12ad P e49c0e2d569b43c9b70ab030d22e63dc: Time spent starting tablet: real 0.010s user 0.010s sys 0.001s
I20260929 20:59:54.771821 4987 ts_tablet_manager.cc:1428] T a2898dd00b974b40912e1f5c1f3f12ad P 49f61222e1564771bb7a79efabec672b: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260929 20:59:54.775018 4989 raft_consensus.cc:348] T a2898dd00b974b40912e1f5c1f3f12ad P 7827899f0e3240b0967165ac05e56cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49c0e2d569b43c9b70ab030d22e63dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40777 } } peers { permanent_uuid: "49f61222e1564771bb7a79efabec672b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37871 } } peers { permanent_uuid: "7827899f0e3240b0967165ac05e56cde" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35337 } }
I20260929 20:59:54.775357 4989 raft_consensus.cc:374] T a2898dd00b974b40912e1f5c1f3f12ad P 7827899f0e3240b0967165ac05e56cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:54.775471 4989 raft_consensus.cc:729] T a2898dd00b974b40912e1f5c1f3f12ad P 7827899f0e3240b0967165ac05e56cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7827899f0e3240b0967165ac05e56cde, State: Initialized, Role: FOLLOWER
I20260929 20:59:54.775945 4989 consensus_queue.cc:260] T a2898dd00b974b40912e1f5c1f3f12ad P 7827899f0e3240b0967165ac05e56cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49c0e2d569b43c9b70ab030d22e63dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40777 } } peers { permanent_uuid: "49f61222e1564771bb7a79efabec672b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37871 } } peers { permanent_uuid: "7827899f0e3240b0967165ac05e56cde" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35337 } }
I20260929 20:59:54.776906 4837 heartbeater.cc:503] Master 127.0.97.62:46399 was elected leader, sending a full tablet report...
I20260929 20:59:54.777460 4989 ts_tablet_manager.cc:1428] T a2898dd00b974b40912e1f5c1f3f12ad P 7827899f0e3240b0967165ac05e56cde: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
W20260929 20:59:54.897236 4970 tablet.cc:2369] T a2898dd00b974b40912e1f5c1f3f12ad P e49c0e2d569b43c9b70ab030d22e63dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:59:54.906802 4993 raft_consensus.cc:482] T a2898dd00b974b40912e1f5c1f3f12ad P e49c0e2d569b43c9b70ab030d22e63dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:59:54.907155 4993 raft_consensus.cc:504] T a2898dd00b974b40912e1f5c1f3f12ad P e49c0e2d569b43c9b70ab030d22e63dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49c0e2d569b43c9b70ab030d22e63dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40777 } } peers { permanent_uuid: "49f61222e1564771bb7a79efabec672b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37871 } } peers { permanent_uuid: "7827899f0e3240b0967165ac05e56cde" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35337 } }
I20260929 20:59:54.908897 4993 leader_election.cc:288] T a2898dd00b974b40912e1f5c1f3f12ad P e49c0e2d569b43c9b70ab030d22e63dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49f61222e1564771bb7a79efabec672b (127.0.97.1:37871), 7827899f0e3240b0967165ac05e56cde (127.0.97.2:35337)
I20260929 20:59:54.915153 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:54.909281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37871 (local address 127.0.97.3:57763)
0929 20:59:54.909504 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:54.909525 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:54.909645 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:54.910170 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:54.910174 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:54.910194 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:59:54.910356 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:54.910363 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:54.911655 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:54.911661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:54.913236 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:54.913247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:54.914556 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:54.914561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:54.914694 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:54.914731 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 20:59:54.914807 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 20:59:54.914895 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":1}
I20260929 20:59:54.915237 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:54.909376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57763 (local address 127.0.97.1:37871)
0929 20:59:54.909559 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:54.909563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:54.909581 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:59:54.909814 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:54.909819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:54.909895 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:54.909992 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:59:54.910524 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:54.911540 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:54.913400 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:54.914382 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:54.914515 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:54.914613 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:54.914858 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 20:59:54.914985 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 20:59:54.916103 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:54.909874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58237 (local address 127.0.97.2:35337)
0929 20:59:54.910053 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:54.910058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:54.910457 (+ 399us) server_negotiation.cc:407] Connection header received
0929 20:59:54.910604 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:54.910607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:54.910659 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:54.910729 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:59:54.911273 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:54.912172 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:54.913926 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:54.915081 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:54.915208 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:54.915308 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:54.915740 (+ 432us) server_negotiation.cc:299] Negotiation successful
0929 20:59:54.915936 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:59:54.916113 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:54.909686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35337 (local address 127.0.97.3:58237)
0929 20:59:54.910350 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:54.910363 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:54.910481 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:54.910775 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:54.910779 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:54.910805 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:59:54.911119 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:54.911128 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:54.912336 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:54.912339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:54.913778 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:54.913788 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:54.915453 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:54.915458 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:54.915591 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:54.915612 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 20:59:54.915683 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 20:59:54.915768 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":395,"threads_started":1}
I20260929 20:59:54.916347 4659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2898dd00b974b40912e1f5c1f3f12ad" candidate_uuid: "e49c0e2d569b43c9b70ab030d22e63dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49f61222e1564771bb7a79efabec672b" is_pre_election: true
I20260929 20:59:54.916940 4659 raft_consensus.cc:2454] T a2898dd00b974b40912e1f5c1f3f12ad P 49f61222e1564771bb7a79efabec672b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e49c0e2d569b43c9b70ab030d22e63dc in term 0.
W20260929 20:59:54.917096 4838 tablet.cc:2369] T a2898dd00b974b40912e1f5c1f3f12ad P 7827899f0e3240b0967165ac05e56cde: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:59:54.917073 4792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2898dd00b974b40912e1f5c1f3f12ad" candidate_uuid: "e49c0e2d569b43c9b70ab030d22e63dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7827899f0e3240b0967165ac05e56cde" is_pre_election: true
I20260929 20:59:54.917649 4792 raft_consensus.cc:2454] T a2898dd00b974b40912e1f5c1f3f12ad P 7827899f0e3240b0967165ac05e56cde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e49c0e2d569b43c9b70ab030d22e63dc in term 0.
I20260929 20:59:54.917733 4858 leader_election.cc:302] T a2898dd00b974b40912e1f5c1f3f12ad P e49c0e2d569b43c9b70ab030d22e63dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49f61222e1564771bb7a79efabec672b, e49c0e2d569b43c9b70ab030d22e63dc; no voters:
I20260929 20:59:54.918166 4993 raft_consensus.cc:2781] T a2898dd00b974b40912e1f5c1f3f12ad P e49c0e2d569b43c9b70ab030d22e63dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:59:54.918330 4993 raft_consensus.cc:482] T a2898dd00b974b40912e1f5c1f3f12ad P e49c0e2d569b43c9b70ab030d22e63dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:59:54.918439 4993 raft_consensus.cc:3037] T a2898dd00b974b40912e1f5c1f3f12ad P e49c0e2d569b43c9b70ab030d22e63dc [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:54.920605 4993 raft_consensus.cc:504] T a2898dd00b974b40912e1f5c1f3f12ad P e49c0e2d569b43c9b70ab030d22e63dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49c0e2d569b43c9b70ab030d22e63dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40777 } } peers { permanent_uuid: "49f61222e1564771bb7a79efabec672b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37871 } } peers { permanent_uuid: "7827899f0e3240b0967165ac05e56cde" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35337 } }
I20260929 20:59:54.921209 4993 leader_election.cc:288] T a2898dd00b974b40912e1f5c1f3f12ad P e49c0e2d569b43c9b70ab030d22e63dc [CANDIDATE]: Term 1 election: Requested vote from peers 49f61222e1564771bb7a79efabec672b (127.0.97.1:37871), 7827899f0e3240b0967165ac05e56cde (127.0.97.2:35337)
I20260929 20:59:54.921690 4659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2898dd00b974b40912e1f5c1f3f12ad" candidate_uuid: "e49c0e2d569b43c9b70ab030d22e63dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49f61222e1564771bb7a79efabec672b"
I20260929 20:59:54.921866 4792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2898dd00b974b40912e1f5c1f3f12ad" candidate_uuid: "e49c0e2d569b43c9b70ab030d22e63dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7827899f0e3240b0967165ac05e56cde"
I20260929 20:59:54.921895 4659 raft_consensus.cc:3037] T a2898dd00b974b40912e1f5c1f3f12ad P 49f61222e1564771bb7a79efabec672b [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:54.922096 4792 raft_consensus.cc:3037] T a2898dd00b974b40912e1f5c1f3f12ad P 7827899f0e3240b0967165ac05e56cde [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:54.924052 4659 raft_consensus.cc:2454] T a2898dd00b974b40912e1f5c1f3f12ad P 49f61222e1564771bb7a79efabec672b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e49c0e2d569b43c9b70ab030d22e63dc in term 1.
I20260929 20:59:54.924196 4792 raft_consensus.cc:2454] T a2898dd00b974b40912e1f5c1f3f12ad P 7827899f0e3240b0967165ac05e56cde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e49c0e2d569b43c9b70ab030d22e63dc in term 1.
I20260929 20:59:54.924472 4858 leader_election.cc:302] T a2898dd00b974b40912e1f5c1f3f12ad P e49c0e2d569b43c9b70ab030d22e63dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49f61222e1564771bb7a79efabec672b, e49c0e2d569b43c9b70ab030d22e63dc; no voters:
I20260929 20:59:54.924743 4993 raft_consensus.cc:2781] T a2898dd00b974b40912e1f5c1f3f12ad P e49c0e2d569b43c9b70ab030d22e63dc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:59:54.925182 4993 raft_consensus.cc:686] T a2898dd00b974b40912e1f5c1f3f12ad P e49c0e2d569b43c9b70ab030d22e63dc [term 1 LEADER]: Becoming Leader. State: Replica: e49c0e2d569b43c9b70ab030d22e63dc, State: Running, Role: LEADER
I20260929 20:59:54.925678 4993 consensus_queue.cc:237] T a2898dd00b974b40912e1f5c1f3f12ad P e49c0e2d569b43c9b70ab030d22e63dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49c0e2d569b43c9b70ab030d22e63dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40777 } } peers { permanent_uuid: "49f61222e1564771bb7a79efabec672b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37871 } } peers { permanent_uuid: "7827899f0e3240b0967165ac05e56cde" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35337 } }
I20260929 20:59:54.930222 4516 catalog_manager.cc:5495] T a2898dd00b974b40912e1f5c1f3f12ad P e49c0e2d569b43c9b70ab030d22e63dc reported cstate change: term changed from 0 to 1, leader changed from <none> to e49c0e2d569b43c9b70ab030d22e63dc (127.0.97.3). New cstate: current_term: 1 leader_uuid: "e49c0e2d569b43c9b70ab030d22e63dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49c0e2d569b43c9b70ab030d22e63dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49f61222e1564771bb7a79efabec672b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7827899f0e3240b0967165ac05e56cde" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35337 } health_report { overall_health: UNKNOWN } } }
W20260929 20:59:54.941679 4705 tablet.cc:2369] T a2898dd00b974b40912e1f5c1f3f12ad P 49f61222e1564771bb7a79efabec672b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:59:54.965346 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:54.959770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48720 (local address 127.0.97.3:40777)
0929 20:59:54.959979 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:54.959985 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:54.960059 (+ 74us) server_negotiation.cc:407] Connection header received
0929 20:59:54.960312 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:54.960336 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:54.960446 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:54.960595 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:59:54.961105 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:54.961989 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:54.963257 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:54.963625 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:54.963724 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:54.964236 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:59:54.964732 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:59:54.964914 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:54.964981 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 20:59:54.965044 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 20:59:54.973039 4792 raft_consensus.cc:1261] T a2898dd00b974b40912e1f5c1f3f12ad P 7827899f0e3240b0967165ac05e56cde [term 1 FOLLOWER]: Refusing update from remote peer e49c0e2d569b43c9b70ab030d22e63dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:59:54.973068 4659 raft_consensus.cc:1261] T a2898dd00b974b40912e1f5c1f3f12ad P 49f61222e1564771bb7a79efabec672b [term 1 FOLLOWER]: Refusing update from remote peer e49c0e2d569b43c9b70ab030d22e63dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:59:54.973960 4997 consensus_queue.cc:1035] T a2898dd00b974b40912e1f5c1f3f12ad P e49c0e2d569b43c9b70ab030d22e63dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7827899f0e3240b0967165ac05e56cde" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:59:54.974313 4993 consensus_queue.cc:1035] T a2898dd00b974b40912e1f5c1f3f12ad P e49c0e2d569b43c9b70ab030d22e63dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "49f61222e1564771bb7a79efabec672b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:59:54.983572 5000 mvcc.cc:205] Tried to move back new op lower bound from 7334771076999737344 to 7334771076819320832. Current Snapshot: MvccSnapshot[applied={T|T < 7334771076999737344}]
I20260929 20:59:54.984961 5001 mvcc.cc:205] Tried to move back new op lower bound from 7334771076999737344 to 7334771076819320832. Current Snapshot: MvccSnapshot[applied={T|T < 7334771076999737344}]
I20260929 20:59:54.998137 4516 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:59:54.998397 4516 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:59:55.001451 4516 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb04e3d3d3734e84a870b8cff0c5cc3a: Sending DeleteTablet for 3 replicas of tablet a2898dd00b974b40912e1f5c1f3f12ad
I20260929 20:59:55.002401 4904 tablet_service.cc:1514] Processing DeleteTablet for tablet a2898dd00b974b40912e1f5c1f3f12ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:59:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48704
I20260929 20:59:55.002650 4772 tablet_service.cc:1514] Processing DeleteTablet for tablet a2898dd00b974b40912e1f5c1f3f12ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:59:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44706
I20260929 20:59:55.003074 4639 tablet_service.cc:1514] Processing DeleteTablet for tablet a2898dd00b974b40912e1f5c1f3f12ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:59:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55934
I20260929 20:59:55.003551 5011 tablet_replica.cc:330] T a2898dd00b974b40912e1f5c1f3f12ad P 7827899f0e3240b0967165ac05e56cde: stopping tablet replica
I20260929 20:59:55.004019 5011 raft_consensus.cc:2229] T a2898dd00b974b40912e1f5c1f3f12ad P 7827899f0e3240b0967165ac05e56cde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:59:55.004391 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 4577
I20260929 20:59:55.004629 5011 raft_consensus.cc:2258] T a2898dd00b974b40912e1f5c1f3f12ad P 7827899f0e3240b0967165ac05e56cde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:59:55.006882 5011 ts_tablet_manager.cc:1905] T a2898dd00b974b40912e1f5c1f3f12ad P 7827899f0e3240b0967165ac05e56cde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:59:55.015934 5011 ts_tablet_manager.cc:1918] T a2898dd00b974b40912e1f5c1f3f12ad P 7827899f0e3240b0967165ac05e56cde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:59:55.016180 5011 log.cc:1196] T a2898dd00b974b40912e1f5c1f3f12ad P 7827899f0e3240b0967165ac05e56cde: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-1/wal/wals/a2898dd00b974b40912e1f5c1f3f12ad
I20260929 20:59:55.016819 5011 ts_tablet_manager.cc:1939] T a2898dd00b974b40912e1f5c1f3f12ad P 7827899f0e3240b0967165ac05e56cde: Deleting consensus metadata
I20260929 20:59:55.018870 4501 catalog_manager.cc:4841] TS 7827899f0e3240b0967165ac05e56cde (127.0.97.2:35337): tablet a2898dd00b974b40912e1f5c1f3f12ad (table test-table [id=0ad6737b25444202a5b78e4234e521d6]) successfully deleted
Sep 29 20:59:55 dist-test-slave-1dms krb5kdc[4472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:59:55.025188 5012 tablet_replica.cc:330] T a2898dd00b974b40912e1f5c1f3f12ad P 49f61222e1564771bb7a79efabec672b: stopping tablet replica
I20260929 20:59:55.025692 5012 raft_consensus.cc:2229] T a2898dd00b974b40912e1f5c1f3f12ad P 49f61222e1564771bb7a79efabec672b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:59:55.027053 5010 tablet_replica.cc:330] T a2898dd00b974b40912e1f5c1f3f12ad P e49c0e2d569b43c9b70ab030d22e63dc: stopping tablet replica
I20260929 20:59:55.027493 5010 raft_consensus.cc:2229] T a2898dd00b974b40912e1f5c1f3f12ad P e49c0e2d569b43c9b70ab030d22e63dc [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:59:55.028146 5010 raft_consensus.cc:2258] T a2898dd00b974b40912e1f5c1f3f12ad P e49c0e2d569b43c9b70ab030d22e63dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:59:55.026264 5012 raft_consensus.cc:2258] T a2898dd00b974b40912e1f5c1f3f12ad P 49f61222e1564771bb7a79efabec672b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:59:55.030531 5012 ts_tablet_manager.cc:1905] T a2898dd00b974b40912e1f5c1f3f12ad P 49f61222e1564771bb7a79efabec672b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:59:55.030539 5010 ts_tablet_manager.cc:1905] T a2898dd00b974b40912e1f5c1f3f12ad P e49c0e2d569b43c9b70ab030d22e63dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:59:55.036949 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:55.005853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55946 (local address 127.0.97.1:37871)
0929 20:59:55.006033 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:55.006040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:55.015859 (+ 9819us) server_negotiation.cc:407] Connection header received
0929 20:59:55.016335 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:55.016344 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:55.016459 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:55.016540 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:55.018275 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:55.019220 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:55.020463 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:55.021133 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:55.021274 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:55.025499 (+ 4225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:55.025511 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:55.025529 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:55.025579 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:55.029755 (+ 4176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:55.033652 (+ 3897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:55.033661 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:55.033665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:55.033848 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:55.034325 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:55.034331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:55.034334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:55.034783 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:55.035132 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:55.036290 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0929 20:59:55.036507 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 20:59:55.037700 5012 ts_tablet_manager.cc:1918] T a2898dd00b974b40912e1f5c1f3f12ad P 49f61222e1564771bb7a79efabec672b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:59:55.037810 5010 ts_tablet_manager.cc:1918] T a2898dd00b974b40912e1f5c1f3f12ad P e49c0e2d569b43c9b70ab030d22e63dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:59:55.037895 5012 log.cc:1196] T a2898dd00b974b40912e1f5c1f3f12ad P 49f61222e1564771bb7a79efabec672b: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-0/wal/wals/a2898dd00b974b40912e1f5c1f3f12ad
I20260929 20:59:55.037943 5010 log.cc:1196] T a2898dd00b974b40912e1f5c1f3f12ad P e49c0e2d569b43c9b70ab030d22e63dc: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790715556572573-388-0/minicluster-data/ts-2/wal/wals/a2898dd00b974b40912e1f5c1f3f12ad
I20260929 20:59:55.037923 4699 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:59:55.038514 5010 ts_tablet_manager.cc:1939] T a2898dd00b974b40912e1f5c1f3f12ad P e49c0e2d569b43c9b70ab030d22e63dc: Deleting consensus metadata
I20260929 20:59:55.040354 4501 catalog_manager.cc:4841] TS e49c0e2d569b43c9b70ab030d22e63dc (127.0.97.3:40777): tablet a2898dd00b974b40912e1f5c1f3f12ad (table test-table [id=0ad6737b25444202a5b78e4234e521d6]) successfully deleted
I20260929 20:59:55.126628 5012 ts_tablet_manager.cc:1939] T a2898dd00b974b40912e1f5c1f3f12ad P 49f61222e1564771bb7a79efabec672b: Deleting consensus metadata
I20260929 20:59:55.126811 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 4577
W20260929 20:59:55.149613 4501 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:37871 (error 108)
I20260929 20:59:55.149880 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 4710
I20260929 20:59:55.151690 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:55.151223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37871 (local address 127.0.0.1:55956)
0929 20:59:55.151406 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:55.151525 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260929 20:59:55.152033 4501 catalog_manager.cc:4584] TS 49f61222e1564771bb7a79efabec672b (127.0.97.1:37871): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a2898dd00b974b40912e1f5c1f3f12ad: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37871: connect: Connection refused (error 111)
Sep 29 20:59:55 dist-test-slave-1dms krb5kdc[4472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:59:55.166144 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:55.151334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44716 (local address 127.0.97.2:35337)
0929 20:59:55.151502 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:55.151508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:55.151527 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:59:55.151592 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:55.151597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:55.151702 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:55.151806 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:55.153221 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:55.154067 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:55.155265 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:55.155643 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:55.155778 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:55.160122 (+ 4344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:55.160136 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:55.160151 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:55.160191 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:55.163083 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:55.163808 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:55.163817 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:55.163819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:55.163948 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:55.164437 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:55.164442 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:55.164445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:55.164938 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:55.165251 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:55.165555 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 20:59:55.165788 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:59:55.169531 4832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:59:55.205910 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:55.205280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37871 (local address 127.0.0.1:55960)
0929 20:59:55.205564 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:55.205711 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260929 20:59:55.207087 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:55.206693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37871 (local address 127.0.0.1:55970)
0929 20:59:55.206851 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:55.206947 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 20:59:55.262693 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 4710
I20260929 20:59:55.267906 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:55.267389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37871 (local address 127.0.0.1:55984)
0929 20:59:55.267625 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:55.267729 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 20:59:55.269142 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:55.268663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37871 (local address 127.0.0.1:55998)
0929 20:59:55.268881 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:55.268991 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 20:59:55.280610 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 4842
Sep 29 20:59:55 dist-test-slave-1dms krb5kdc[4472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:59:55.297071 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:55.281410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48734 (local address 127.0.97.3:40777)
0929 20:59:55.281602 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:55.281608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:55.281631 (+ 23us) server_negotiation.cc:407] Connection header received
0929 20:59:55.281771 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:55.281778 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:55.281873 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:55.281989 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:55.283180 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:55.284375 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:55.285779 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:55.286493 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:55.286635 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:55.291154 (+ 4519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:55.291165 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:55.291181 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:59:55.291226 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:55.294302 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:55.295082 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:55.295094 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:55.295097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:55.295247 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:59:55.295639 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:59:55.295642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:59:55.295644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:59:55.295967 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:55.296190 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:55.296526 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 20:59:55.296674 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 20:59:55.297830 4964 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:59:55.349603 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:55.349051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37871 (local address 127.0.0.1:56008)
0929 20:59:55.349290 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:55.349401 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 20:59:55.350675 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:55.350270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37871 (local address 127.0.0.1:56012)
0929 20:59:55.350458 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:55.350541 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260929 20:59:55.418375 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 4842
I20260929 20:59:55.437067 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 4485
I20260929 20:59:55.438318 4546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:59:55.546078 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 4485
I20260929 20:59:55.546208 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:55.545539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37871 (local address 127.0.0.1:56016)
0929 20:59:55.545822 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:55.545945 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
2026-09-29T20:59:55Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4186 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T20:59:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:59:55Z Disabled control of system clock
I20260929 20:59:55.610775 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35037
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:33321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 20:59:55.921880 5022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:55.922164 5022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:55.922246 5022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:55.929402 5022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:59:55.929510 5022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:55.929550 5022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:55.929606 5022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:59:55.929649 5022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:59:55.938688 5022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35037
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:55.939913 5022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:55.941697 5022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:55.953588 5027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:55.953567 5028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:55.954030 5030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:55.954293 5022 server_base.cc:1034] running on GCE node
I20260929 20:59:55.955114 5022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:55.956630 5022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:55.957867 5022 hybrid_clock.cc:648] HybridClock initialized: now 1790715595957808 us; error 61 us; skew 500 ppm
I20260929 20:59:55.958235 5022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:59:55.960539 5022 webserver.cc:460] Webserver started at http://127.0.97.62:45303/ using document root <none> and password file <none>
I20260929 20:59:55.961315 5022 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:55.961447 5022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:55.961777 5022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:55.964707 5022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "9135f76961884c7aaac3f07577409849"
format_stamp: "Formatted at 2026-09-29 20:59:55 on dist-test-slave-1dms"
server_key: "7dcaceac3fe2e86c4c3c9b91abdc797d"
server_key_iv: "7f14dd59e500f21e12a533167f87d6d6"
server_key_version: "encryptionkey@0"
I20260929 20:59:55.965734 5022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "9135f76961884c7aaac3f07577409849"
format_stamp: "Formatted at 2026-09-29 20:59:55 on dist-test-slave-1dms"
server_key: "7dcaceac3fe2e86c4c3c9b91abdc797d"
server_key_iv: "7f14dd59e500f21e12a533167f87d6d6"
server_key_version: "encryptionkey@0"
I20260929 20:59:55.971122 5022 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:59:55.974570 5036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:55.975893 5022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:59:55.976065 5022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "9135f76961884c7aaac3f07577409849"
format_stamp: "Formatted at 2026-09-29 20:59:55 on dist-test-slave-1dms"
server_key: "7dcaceac3fe2e86c4c3c9b91abdc797d"
server_key_iv: "7f14dd59e500f21e12a533167f87d6d6"
server_key_version: "encryptionkey@0"
I20260929 20:59:55.976275 5022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:55.982031 5022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:55.982735 5022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:55.983023 5022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:56.000497 5022 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35037
I20260929 20:59:56.000527 5087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35037 every 8 connection(s)
I20260929 20:59:56.001789 5022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:59:56.005316 5088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:56.006614 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 5022
I20260929 20:59:56.006793 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 20:59:56.007452 388 external_mini_cluster.cc:1428] Setting key 57e0e48615c8c2466616b1bb81f65357
I20260929 20:59:56.016988 5091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:56.009444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35230 (local address 127.0.97.62:35037)
0929 20:59:56.010533 (+ 1089us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:56.010556 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:56.010610 (+ 54us) server_negotiation.cc:407] Connection header received
0929 20:59:56.012869 (+ 2259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:56.012897 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:56.013397 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:56.013944 (+ 547us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:56.014942 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:56.014950 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:56.014972 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:59:56.014989 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:56.015061 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:59:56.015087 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:56.015316 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:56.015458 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 20:59:56.015725 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":348,"threads_started":1}
I20260929 20:59:56.018682 5088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9135f76961884c7aaac3f07577409849: Bootstrap starting.
I20260929 20:59:56.021562 5088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9135f76961884c7aaac3f07577409849: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:56.022504 5088 log.cc:824] T 00000000000000000000000000000000 P 9135f76961884c7aaac3f07577409849: Log is configured to *not* fsync() on all Append() calls
I20260929 20:59:56.025076 5088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9135f76961884c7aaac3f07577409849: No bootstrap required, opened a new log
I20260929 20:59:56.031917 5088 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9135f76961884c7aaac3f07577409849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9135f76961884c7aaac3f07577409849" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35037 } }
I20260929 20:59:56.032253 5088 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9135f76961884c7aaac3f07577409849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:56.032366 5088 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9135f76961884c7aaac3f07577409849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9135f76961884c7aaac3f07577409849, State: Initialized, Role: FOLLOWER
I20260929 20:59:56.032954 5088 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9135f76961884c7aaac3f07577409849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9135f76961884c7aaac3f07577409849" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35037 } }
I20260929 20:59:56.033178 5088 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9135f76961884c7aaac3f07577409849 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:59:56.033304 5088 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9135f76961884c7aaac3f07577409849 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:59:56.033474 5088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9135f76961884c7aaac3f07577409849 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:56.035513 5088 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9135f76961884c7aaac3f07577409849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9135f76961884c7aaac3f07577409849" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35037 } }
I20260929 20:59:56.035998 5088 leader_election.cc:302] T 00000000000000000000000000000000 P 9135f76961884c7aaac3f07577409849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9135f76961884c7aaac3f07577409849; no voters:
I20260929 20:59:56.036535 5088 leader_election.cc:288] T 00000000000000000000000000000000 P 9135f76961884c7aaac3f07577409849 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:59:56.036739 5093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9135f76961884c7aaac3f07577409849 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:59:56.037460 5093 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9135f76961884c7aaac3f07577409849 [term 1 LEADER]: Becoming Leader. State: Replica: 9135f76961884c7aaac3f07577409849, State: Running, Role: LEADER
I20260929 20:59:56.037997 5093 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9135f76961884c7aaac3f07577409849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9135f76961884c7aaac3f07577409849" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35037 } }
I20260929 20:59:56.038775 5088 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9135f76961884c7aaac3f07577409849 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:59:56.041041 5094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9135f76961884c7aaac3f07577409849 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9135f76961884c7aaac3f07577409849" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9135f76961884c7aaac3f07577409849" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35037 } } }
I20260929 20:59:56.041406 5094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9135f76961884c7aaac3f07577409849 [sys.catalog]: This master's current role is: LEADER
I20260929 20:59:56.042583 5095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9135f76961884c7aaac3f07577409849 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9135f76961884c7aaac3f07577409849. Latest consensus state: current_term: 1 leader_uuid: "9135f76961884c7aaac3f07577409849" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9135f76961884c7aaac3f07577409849" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35037 } } }
I20260929 20:59:56.042912 5095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9135f76961884c7aaac3f07577409849 [sys.catalog]: This master's current role is: LEADER
I20260929 20:59:56.044132 5101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:59:56.048420 5101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:59:56.056401 5101 catalog_manager.cc:1347] Generated new cluster ID: 8ace9e06a2f34c82b0d9b2ae5fe08b70
I20260929 20:59:56.056504 5101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:59:56.066239 5101 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:59:56.066596 5101 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:59:56.074309 5101 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9135f76961884c7aaac3f07577409849: Generated new TSK 0
I20260929 20:59:56.075073 5101 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:59:56.090683 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35037
--builtin_ntp_servers=127.0.97.20:33321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 20:59:56.413174 5112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:56.413439 5112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:56.413594 5112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:56.420938 5112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:56.421038 5112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:56.421257 5112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:59:56.430565 5112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35037
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:56.431910 5112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:56.433748 5112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:56.446919 5120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:56.446959 5118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:56.446987 5117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:56.448093 5112 server_base.cc:1034] running on GCE node
I20260929 20:59:56.448729 5112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:56.449739 5112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:56.450937 5112 hybrid_clock.cc:648] HybridClock initialized: now 1790715596450895 us; error 37 us; skew 500 ppm
I20260929 20:59:56.451260 5112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:59:56.457363 5112 webserver.cc:460] Webserver started at http://127.0.97.1:33235/ using document root <none> and password file <none>
I20260929 20:59:56.458150 5112 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:56.458240 5112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:56.458590 5112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:56.461504 5112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-0/data/instance:
uuid: "80e2651f85f241ef8b701cb1e78e3f1c"
format_stamp: "Formatted at 2026-09-29 20:59:56 on dist-test-slave-1dms"
server_key: "e54ce862d2742dd003d971fcb3d22565"
server_key_iv: "d2cca5c3b8d483a53f85b589cf161e01"
server_key_version: "encryptionkey@0"
I20260929 20:59:56.462283 5112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-0/wal/instance:
uuid: "80e2651f85f241ef8b701cb1e78e3f1c"
format_stamp: "Formatted at 2026-09-29 20:59:56 on dist-test-slave-1dms"
server_key: "e54ce862d2742dd003d971fcb3d22565"
server_key_iv: "d2cca5c3b8d483a53f85b589cf161e01"
server_key_version: "encryptionkey@0"
I20260929 20:59:56.467149 5112 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260929 20:59:56.470482 5126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:56.471750 5112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:59:56.471910 5112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-0/wal
uuid: "80e2651f85f241ef8b701cb1e78e3f1c"
format_stamp: "Formatted at 2026-09-29 20:59:56 on dist-test-slave-1dms"
server_key: "e54ce862d2742dd003d971fcb3d22565"
server_key_iv: "d2cca5c3b8d483a53f85b589cf161e01"
server_key_version: "encryptionkey@0"
I20260929 20:59:56.472136 5112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:56.477568 5112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:56.478359 5112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:56.478610 5112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:56.479619 5112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:59:56.481138 5112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:59:56.481266 5112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:56.481371 5112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:59:56.481431 5112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:56.519939 5112 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45031
I20260929 20:59:56.519985 5238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45031 every 8 connection(s)
I20260929 20:59:56.521334 5112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:59:56.529325 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 5112
I20260929 20:59:56.529561 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:59:56.529618 5241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:56.523864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36273 (local address 127.0.97.62:35037)
0929 20:59:56.524310 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:56.524315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:56.525562 (+ 1247us) server_negotiation.cc:407] Connection header received
0929 20:59:56.527235 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:56.527241 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:56.527306 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:56.527430 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:56.528746 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:56.528751 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:56.528756 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:59:56.528798 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:56.528881 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:59:56.528897 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:56.529009 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:56.529235 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 20:59:56.529323 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":239,"threads_started":1}
I20260929 20:59:56.530234 388 external_mini_cluster.cc:1428] Setting key cf66c248f85e07fa29f35bd699f80f4f
I20260929 20:59:56.530608 5242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:56.524481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.1:36273)
0929 20:59:56.525409 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:56.525449 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:56.526977 (+ 1528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:56.527663 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:56.527670 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:56.527764 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:56.528450 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:59:56.528453 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:56.528488 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:59:56.528569 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:56.529004 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:56.529014 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:56.529023 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 20:59:56.529370 (+ 347us) client_negotiation.cc:240] Negotiation successful
0929 20:59:56.529637 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":307,"threads_started":1}
I20260929 20:59:56.532289 5239 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35037
I20260929 20:59:56.532584 5239 heartbeater.cc:464] Registering TS with master...
I20260929 20:59:56.533087 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35037
--builtin_ntp_servers=127.0.97.20:33321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 20:59:56.533238 5239 heartbeater.cc:511] Master 127.0.97.62:35037 requested a full tablet report, sending...
I20260929 20:59:56.535151 5053 ts_manager.cc:194] Registered new tserver with Master: 80e2651f85f241ef8b701cb1e78e3f1c (127.0.97.1:45031)
W20260929 20:59:56.848246 5243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:56.848512 5243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:56.848672 5243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:56.855774 5243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:56.855861 5243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:56.856034 5243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:59:56.864910 5243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35037
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:56.866174 5243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:56.867892 5243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:56.880414 5248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:56.880364 5249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:56.880438 5251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:56.881832 5243 server_base.cc:1034] running on GCE node
I20260929 20:59:56.882387 5243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:56.883277 5243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:56.884471 5243 hybrid_clock.cc:648] HybridClock initialized: now 1790715596884414 us; error 64 us; skew 500 ppm
I20260929 20:59:56.884760 5243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:59:56.886955 5243 webserver.cc:460] Webserver started at http://127.0.97.2:34311/ using document root <none> and password file <none>
I20260929 20:59:56.887641 5243 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:56.887745 5243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:56.888005 5243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:56.890820 5243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-1/data/instance:
uuid: "a2a3c0bbf89f417c9bd7bcd3aa9ed6d9"
format_stamp: "Formatted at 2026-09-29 20:59:56 on dist-test-slave-1dms"
server_key: "2f3010f8d4a1bdb3b9534279b5180da0"
server_key_iv: "436847339387b1665abc88b89f60df92"
server_key_version: "encryptionkey@0"
I20260929 20:59:56.891522 5243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a2a3c0bbf89f417c9bd7bcd3aa9ed6d9"
format_stamp: "Formatted at 2026-09-29 20:59:56 on dist-test-slave-1dms"
server_key: "2f3010f8d4a1bdb3b9534279b5180da0"
server_key_iv: "436847339387b1665abc88b89f60df92"
server_key_version: "encryptionkey@0"
I20260929 20:59:56.896314 5243 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260929 20:59:56.899708 5257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:56.901069 5243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:59:56.901225 5243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-1/wal
uuid: "a2a3c0bbf89f417c9bd7bcd3aa9ed6d9"
format_stamp: "Formatted at 2026-09-29 20:59:56 on dist-test-slave-1dms"
server_key: "2f3010f8d4a1bdb3b9534279b5180da0"
server_key_iv: "436847339387b1665abc88b89f60df92"
server_key_version: "encryptionkey@0"
I20260929 20:59:56.901444 5243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:56.906862 5243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:56.907610 5243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:56.907887 5243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:56.908751 5243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:59:56.910082 5243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:59:56.910192 5243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:56.910318 5243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:59:56.910406 5243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:56.942474 5243 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45973
I20260929 20:59:56.942516 5369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45973 every 8 connection(s)
I20260929 20:59:56.943873 5243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:59:56.950783 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 5243
I20260929 20:59:56.950960 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:59:56.951381 388 external_mini_cluster.cc:1428] Setting key 051a3ad2fe8b9799937968539f32278a
I20260929 20:59:56.951963 5241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:56.946404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57773 (local address 127.0.97.62:35037)
0929 20:59:56.946567 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:56.946572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:56.947928 (+ 1356us) server_negotiation.cc:407] Connection header received
0929 20:59:56.949658 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:56.949663 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:56.949742 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:56.949852 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:56.951232 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:56.951237 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:56.951242 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:59:56.951260 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:56.951301 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:59:56.951313 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:56.951432 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:56.951618 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 20:59:56.951682 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 20:59:56.953099 5372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:56.946867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.2:57773)
0929 20:59:56.947767 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:56.947802 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:56.949386 (+ 1584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:56.950071 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:56.950079 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:56.950166 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:56.950875 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:59:56.950878 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:56.950944 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:59:56.951065 (+ 121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:56.951403 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:56.951413 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:56.951420 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 20:59:56.951704 (+ 284us) client_negotiation.cc:240] Negotiation successful
0929 20:59:56.951947 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":270,"threads_started":1}
I20260929 20:59:56.954268 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35037
--builtin_ntp_servers=127.0.97.20:33321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 20:59:56.954474 5370 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35037
I20260929 20:59:56.954782 5370 heartbeater.cc:464] Registering TS with master...
I20260929 20:59:56.955346 5370 heartbeater.cc:511] Master 127.0.97.62:35037 requested a full tablet report, sending...
I20260929 20:59:56.956648 5053 ts_manager.cc:194] Registered new tserver with Master: a2a3c0bbf89f417c9bd7bcd3aa9ed6d9 (127.0.97.2:45973)
W20260929 20:59:57.264882 5373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:57.265182 5373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:57.265357 5373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:57.272365 5373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:57.272490 5373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:57.272663 5373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:59:57.281733 5373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35037
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:57.282943 5373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:57.284649 5373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:57.297282 5378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:57.297315 5379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:57.297266 5381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:57.298125 5373 server_base.cc:1034] running on GCE node
I20260929 20:59:57.298698 5373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:57.299628 5373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:57.300941 5373 hybrid_clock.cc:648] HybridClock initialized: now 1790715597300851 us; error 70 us; skew 500 ppm
I20260929 20:59:57.301261 5373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:59:57.303452 5373 webserver.cc:460] Webserver started at http://127.0.97.3:42069/ using document root <none> and password file <none>
I20260929 20:59:57.304178 5373 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:57.304324 5373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:57.304636 5373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:57.307422 5373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-2/data/instance:
uuid: "b77bac84be7640b2b52fcf54d5460e92"
format_stamp: "Formatted at 2026-09-29 20:59:57 on dist-test-slave-1dms"
server_key: "8c7c512118bcdf470ba9205e0734fe37"
server_key_iv: "108fa99af1f50939bfaecf71de1e3fd4"
server_key_version: "encryptionkey@0"
I20260929 20:59:57.308199 5373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b77bac84be7640b2b52fcf54d5460e92"
format_stamp: "Formatted at 2026-09-29 20:59:57 on dist-test-slave-1dms"
server_key: "8c7c512118bcdf470ba9205e0734fe37"
server_key_iv: "108fa99af1f50939bfaecf71de1e3fd4"
server_key_version: "encryptionkey@0"
I20260929 20:59:57.313127 5373 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 20:59:57.316320 5387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:57.317792 5373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:59:57.317966 5373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-2/wal
uuid: "b77bac84be7640b2b52fcf54d5460e92"
format_stamp: "Formatted at 2026-09-29 20:59:57 on dist-test-slave-1dms"
server_key: "8c7c512118bcdf470ba9205e0734fe37"
server_key_iv: "108fa99af1f50939bfaecf71de1e3fd4"
server_key_version: "encryptionkey@0"
I20260929 20:59:57.318188 5373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:57.323663 5373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:57.324390 5373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:57.324659 5373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:57.325567 5373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:59:57.326892 5373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:59:57.327011 5373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:57.327143 5373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:59:57.327234 5373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:57.359305 5373 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37747
I20260929 20:59:57.359357 5499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37747 every 8 connection(s)
I20260929 20:59:57.360633 5373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:59:57.362224 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 5373
I20260929 20:59:57.362390 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:59:57.362959 388 external_mini_cluster.cc:1428] Setting key a6567b0b3296f56d21830a742d1ed41d
I20260929 20:59:57.369529 5241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:57.363602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57443 (local address 127.0.97.62:35037)
0929 20:59:57.363795 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:57.363800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:57.365552 (+ 1752us) server_negotiation.cc:407] Connection header received
0929 20:59:57.367323 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:57.367327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:57.367375 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:57.367459 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:57.368686 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:57.368690 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:57.368693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:59:57.368708 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:57.368739 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:59:57.368751 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:57.368928 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:57.369193 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 20:59:57.369246 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 20:59:57.370445 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:57.364279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.3:57443)
0929 20:59:57.365387 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:57.365440 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:57.367077 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:57.367651 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:57.367658 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:57.367749 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:57.368387 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:59:57.368389 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:57.368426 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:59:57.368508 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:57.368965 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:57.368974 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:57.368981 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 20:59:57.369223 (+ 242us) client_negotiation.cc:240] Negotiation successful
0929 20:59:57.369445 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":315,"threads_started":1}
I20260929 20:59:57.372054 5500 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35037
I20260929 20:59:57.372489 5500 heartbeater.cc:464] Registering TS with master...
I20260929 20:59:57.373180 5500 heartbeater.cc:511] Master 127.0.97.62:35037 requested a full tablet report, sending...
I20260929 20:59:57.374665 5053 ts_manager.cc:194] Registered new tserver with Master: b77bac84be7640b2b52fcf54d5460e92 (127.0.97.3:37747)
I20260929 20:59:57.379503 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:59:57.386699 5241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:57.384289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35246 (local address 127.0.97.62:35037)
0929 20:59:57.384456 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:57.384461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:57.385016 (+ 555us) server_negotiation.cc:407] Connection header received
0929 20:59:57.385156 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:57.385159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:57.385217 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:57.385300 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:57.386269 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:57.386273 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:57.386276 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:59:57.386287 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:57.386312 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:59:57.386322 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:57.386409 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:57.386444 (+ 35us) server_negotiation.cc:299] Negotiation successful
0929 20:59:57.386482 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:59:57.390556 5053 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:59:57.397312 5053 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:59:57.427467 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:57.409178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45973 (local address 127.0.0.1:37412)
0929 20:59:57.421514 (+ 12336us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:57.421579 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:57.422729 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:57.424642 (+ 1913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:57.424648 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:57.424729 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:57.425835 (+ 1106us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:59:57.425841 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:57.425883 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:59:57.425991 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:57.426570 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:57.426581 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:57.426589 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 20:59:57.426790 (+ 201us) client_negotiation.cc:240] Negotiation successful
0929 20:59:57.426896 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11948,"thread_start_us":12621,"threads_started":1}
I20260929 20:59:57.429108 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:57.409817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37412 (local address 127.0.97.2:45973)
0929 20:59:57.423989 (+ 14172us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:57.424002 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:57.424042 (+ 40us) server_negotiation.cc:407] Connection header received
0929 20:59:57.424125 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:57.424135 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:57.424425 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:57.424610 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:57.426180 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:57.426193 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:57.426219 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:59:57.426250 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:57.426315 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:59:57.426339 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:57.426514 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:57.428615 (+ 2101us) server_negotiation.cc:299] Negotiation successful
0929 20:59:57.428733 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13965,"thread_start_us":13800,"threads_started":1}
I20260929 20:59:57.431002 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:57.409806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45031 (local address 127.0.0.1:49094)
0929 20:59:57.428006 (+ 18200us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:57.428029 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:57.428136 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:57.428921 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:57.428926 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:57.428964 (+ 38us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:57.430004 (+ 1040us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:59:57.430008 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:57.430025 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:59:57.430078 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:57.430594 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:57.430601 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:57.430606 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 20:59:57.430696 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 20:59:57.430747 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18099,"thread_start_us":18001,"threads_started":1}
I20260929 20:59:57.431267 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:57.411005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49094 (local address 127.0.97.1:45031)
0929 20:59:57.426784 (+ 15779us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:57.426796 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:57.428257 (+ 1461us) server_negotiation.cc:407] Connection header received
0929 20:59:57.428329 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:57.428336 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:57.428558 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:57.428701 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:57.430267 (+ 1566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:57.430274 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:57.430283 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:59:57.430329 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:57.430393 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:59:57.430414 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:57.430544 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:57.430732 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 20:59:57.430820 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15563,"thread_start_us":15401,"threads_started":1}
I20260929 20:59:57.431574 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:57.409470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37747 (local address 127.0.0.1:45630)
0929 20:59:57.427956 (+ 18486us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:57.427983 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:57.428116 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:57.429823 (+ 1707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:57.429827 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:57.429849 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:57.430436 (+ 587us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:59:57.430441 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:57.430460 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:59:57.430503 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:57.431012 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:57.431017 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:57.431021 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 20:59:57.431161 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 20:59:57.431208 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18302,"thread_start_us":18886,"threads_started":1}
I20260929 20:59:57.432232 5516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:57.410282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45630 (local address 127.0.97.3:37747)
0929 20:59:57.427382 (+ 17100us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:57.427393 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:57.428105 (+ 712us) server_negotiation.cc:407] Connection header received
0929 20:59:57.428328 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:57.428336 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:57.428555 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:57.428704 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:57.430663 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:57.430671 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:57.430682 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:59:57.430701 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:57.430765 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:59:57.430783 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:57.430952 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:57.431776 (+ 824us) server_negotiation.cc:299] Negotiation successful
0929 20:59:57.431863 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16902,"thread_start_us":16688,"threads_started":1}
I20260929 20:59:57.433440 5305 tablet_service.cc:1467] Processing CreateTablet for tablet 3e1074ac828048feac0ab69d9f7287d3 (DEFAULT_TABLE table=test-table [id=85ae27d63d6c4941b0abbfe7301dc655]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:59:57.434840 5305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e1074ac828048feac0ab69d9f7287d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:57.434742 5174 tablet_service.cc:1467] Processing CreateTablet for tablet 3e1074ac828048feac0ab69d9f7287d3 (DEFAULT_TABLE table=test-table [id=85ae27d63d6c4941b0abbfe7301dc655]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:59:57.435544 5435 tablet_service.cc:1467] Processing CreateTablet for tablet 3e1074ac828048feac0ab69d9f7287d3 (DEFAULT_TABLE table=test-table [id=85ae27d63d6c4941b0abbfe7301dc655]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:59:57.436064 5174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e1074ac828048feac0ab69d9f7287d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:57.436888 5435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e1074ac828048feac0ab69d9f7287d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:57.449584 5518 tablet_bootstrap.cc:492] T 3e1074ac828048feac0ab69d9f7287d3 P a2a3c0bbf89f417c9bd7bcd3aa9ed6d9: Bootstrap starting.
I20260929 20:59:57.451639 5520 tablet_bootstrap.cc:492] T 3e1074ac828048feac0ab69d9f7287d3 P b77bac84be7640b2b52fcf54d5460e92: Bootstrap starting.
I20260929 20:59:57.454033 5518 tablet_bootstrap.cc:654] T 3e1074ac828048feac0ab69d9f7287d3 P a2a3c0bbf89f417c9bd7bcd3aa9ed6d9: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:57.455154 5520 tablet_bootstrap.cc:654] T 3e1074ac828048feac0ab69d9f7287d3 P b77bac84be7640b2b52fcf54d5460e92: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:57.455338 5518 log.cc:824] T 3e1074ac828048feac0ab69d9f7287d3 P a2a3c0bbf89f417c9bd7bcd3aa9ed6d9: Log is configured to *not* fsync() on all Append() calls
I20260929 20:59:57.456444 5520 log.cc:824] T 3e1074ac828048feac0ab69d9f7287d3 P b77bac84be7640b2b52fcf54d5460e92: Log is configured to *not* fsync() on all Append() calls
I20260929 20:59:57.457346 5519 tablet_bootstrap.cc:492] T 3e1074ac828048feac0ab69d9f7287d3 P 80e2651f85f241ef8b701cb1e78e3f1c: Bootstrap starting.
I20260929 20:59:57.458678 5518 tablet_bootstrap.cc:492] T 3e1074ac828048feac0ab69d9f7287d3 P a2a3c0bbf89f417c9bd7bcd3aa9ed6d9: No bootstrap required, opened a new log
I20260929 20:59:57.458990 5518 ts_tablet_manager.cc:1397] T 3e1074ac828048feac0ab69d9f7287d3 P a2a3c0bbf89f417c9bd7bcd3aa9ed6d9: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20260929 20:59:57.459369 5520 tablet_bootstrap.cc:492] T 3e1074ac828048feac0ab69d9f7287d3 P b77bac84be7640b2b52fcf54d5460e92: No bootstrap required, opened a new log
I20260929 20:59:57.459626 5520 ts_tablet_manager.cc:1397] T 3e1074ac828048feac0ab69d9f7287d3 P b77bac84be7640b2b52fcf54d5460e92: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 20:59:57.460887 5519 tablet_bootstrap.cc:654] T 3e1074ac828048feac0ab69d9f7287d3 P 80e2651f85f241ef8b701cb1e78e3f1c: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:57.462054 5519 log.cc:824] T 3e1074ac828048feac0ab69d9f7287d3 P 80e2651f85f241ef8b701cb1e78e3f1c: Log is configured to *not* fsync() on all Append() calls
I20260929 20:59:57.465101 5519 tablet_bootstrap.cc:492] T 3e1074ac828048feac0ab69d9f7287d3 P 80e2651f85f241ef8b701cb1e78e3f1c: No bootstrap required, opened a new log
I20260929 20:59:57.465369 5519 ts_tablet_manager.cc:1397] T 3e1074ac828048feac0ab69d9f7287d3 P 80e2651f85f241ef8b701cb1e78e3f1c: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260929 20:59:57.467844 5520 raft_consensus.cc:348] T 3e1074ac828048feac0ab69d9f7287d3 P b77bac84be7640b2b52fcf54d5460e92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a3c0bbf89f417c9bd7bcd3aa9ed6d9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45973 } } peers { permanent_uuid: "b77bac84be7640b2b52fcf54d5460e92" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37747 } } peers { permanent_uuid: "80e2651f85f241ef8b701cb1e78e3f1c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45031 } }
I20260929 20:59:57.468225 5520 raft_consensus.cc:374] T 3e1074ac828048feac0ab69d9f7287d3 P b77bac84be7640b2b52fcf54d5460e92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:57.468319 5520 raft_consensus.cc:729] T 3e1074ac828048feac0ab69d9f7287d3 P b77bac84be7640b2b52fcf54d5460e92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b77bac84be7640b2b52fcf54d5460e92, State: Initialized, Role: FOLLOWER
I20260929 20:59:57.469030 5520 consensus_queue.cc:260] T 3e1074ac828048feac0ab69d9f7287d3 P b77bac84be7640b2b52fcf54d5460e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a3c0bbf89f417c9bd7bcd3aa9ed6d9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45973 } } peers { permanent_uuid: "b77bac84be7640b2b52fcf54d5460e92" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37747 } } peers { permanent_uuid: "80e2651f85f241ef8b701cb1e78e3f1c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45031 } }
I20260929 20:59:57.469117 5518 raft_consensus.cc:348] T 3e1074ac828048feac0ab69d9f7287d3 P a2a3c0bbf89f417c9bd7bcd3aa9ed6d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a3c0bbf89f417c9bd7bcd3aa9ed6d9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45973 } } peers { permanent_uuid: "b77bac84be7640b2b52fcf54d5460e92" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37747 } } peers { permanent_uuid: "80e2651f85f241ef8b701cb1e78e3f1c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45031 } }
I20260929 20:59:57.469527 5518 raft_consensus.cc:374] T 3e1074ac828048feac0ab69d9f7287d3 P a2a3c0bbf89f417c9bd7bcd3aa9ed6d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:57.469614 5518 raft_consensus.cc:729] T 3e1074ac828048feac0ab69d9f7287d3 P a2a3c0bbf89f417c9bd7bcd3aa9ed6d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3c0bbf89f417c9bd7bcd3aa9ed6d9, State: Initialized, Role: FOLLOWER
I20260929 20:59:57.470217 5518 consensus_queue.cc:260] T 3e1074ac828048feac0ab69d9f7287d3 P a2a3c0bbf89f417c9bd7bcd3aa9ed6d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a3c0bbf89f417c9bd7bcd3aa9ed6d9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45973 } } peers { permanent_uuid: "b77bac84be7640b2b52fcf54d5460e92" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37747 } } peers { permanent_uuid: "80e2651f85f241ef8b701cb1e78e3f1c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45031 } }
I20260929 20:59:57.471738 5500 heartbeater.cc:503] Master 127.0.97.62:35037 was elected leader, sending a full tablet report...
I20260929 20:59:57.472229 5370 heartbeater.cc:503] Master 127.0.97.62:35037 was elected leader, sending a full tablet report...
I20260929 20:59:57.472894 5518 ts_tablet_manager.cc:1428] T 3e1074ac828048feac0ab69d9f7287d3 P a2a3c0bbf89f417c9bd7bcd3aa9ed6d9: Time spent starting tablet: real 0.014s user 0.008s sys 0.008s
I20260929 20:59:57.473094 5520 ts_tablet_manager.cc:1428] T 3e1074ac828048feac0ab69d9f7287d3 P b77bac84be7640b2b52fcf54d5460e92: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260929 20:59:57.477104 5519 raft_consensus.cc:348] T 3e1074ac828048feac0ab69d9f7287d3 P 80e2651f85f241ef8b701cb1e78e3f1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a3c0bbf89f417c9bd7bcd3aa9ed6d9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45973 } } peers { permanent_uuid: "b77bac84be7640b2b52fcf54d5460e92" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37747 } } peers { permanent_uuid: "80e2651f85f241ef8b701cb1e78e3f1c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45031 } }
I20260929 20:59:57.477540 5519 raft_consensus.cc:374] T 3e1074ac828048feac0ab69d9f7287d3 P 80e2651f85f241ef8b701cb1e78e3f1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:57.477638 5519 raft_consensus.cc:729] T 3e1074ac828048feac0ab69d9f7287d3 P 80e2651f85f241ef8b701cb1e78e3f1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80e2651f85f241ef8b701cb1e78e3f1c, State: Initialized, Role: FOLLOWER
I20260929 20:59:57.478175 5519 consensus_queue.cc:260] T 3e1074ac828048feac0ab69d9f7287d3 P 80e2651f85f241ef8b701cb1e78e3f1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a3c0bbf89f417c9bd7bcd3aa9ed6d9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45973 } } peers { permanent_uuid: "b77bac84be7640b2b52fcf54d5460e92" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37747 } } peers { permanent_uuid: "80e2651f85f241ef8b701cb1e78e3f1c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45031 } }
I20260929 20:59:57.479112 5239 heartbeater.cc:503] Master 127.0.97.62:35037 was elected leader, sending a full tablet report...
I20260929 20:59:57.479694 5519 ts_tablet_manager.cc:1428] T 3e1074ac828048feac0ab69d9f7287d3 P 80e2651f85f241ef8b701cb1e78e3f1c: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260929 20:59:57.499159 5524 raft_consensus.cc:482] T 3e1074ac828048feac0ab69d9f7287d3 P b77bac84be7640b2b52fcf54d5460e92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:59:57.499378 5524 raft_consensus.cc:504] T 3e1074ac828048feac0ab69d9f7287d3 P b77bac84be7640b2b52fcf54d5460e92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a3c0bbf89f417c9bd7bcd3aa9ed6d9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45973 } } peers { permanent_uuid: "b77bac84be7640b2b52fcf54d5460e92" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37747 } } peers { permanent_uuid: "80e2651f85f241ef8b701cb1e78e3f1c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45031 } }
I20260929 20:59:57.500823 5524 leader_election.cc:288] T 3e1074ac828048feac0ab69d9f7287d3 P b77bac84be7640b2b52fcf54d5460e92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2a3c0bbf89f417c9bd7bcd3aa9ed6d9 (127.0.97.2:45973), 80e2651f85f241ef8b701cb1e78e3f1c (127.0.97.1:45031)
I20260929 20:59:57.504208 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:57.501055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45973 (local address 127.0.97.3:51115)
0929 20:59:57.501303 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:57.501330 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:57.501446 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:57.502457 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:57.502465 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:57.502490 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:57.503266 (+ 776us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:59:57.503269 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:57.503282 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:59:57.503340 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:57.503706 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:57.503733 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:57.503737 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 20:59:57.503858 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 20:59:57.503909 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260929 20:59:57.504931 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:57.501678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38197 (local address 127.0.97.1:45031)
0929 20:59:57.502151 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:57.502158 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:57.502208 (+ 50us) server_negotiation.cc:407] Connection header received
0929 20:59:57.502353 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:57.502374 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:57.502529 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:57.502639 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:57.504139 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:57.504144 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:57.504148 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:59:57.504166 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:57.504201 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:59:57.504232 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:57.504321 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:57.504532 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 20:59:57.504620 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204}
I20260929 20:59:57.504968 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:57.501622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51115 (local address 127.0.97.2:45973)
0929 20:59:57.502104 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:57.502110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:57.502128 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:59:57.502185 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:57.502191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:57.502266 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:57.502362 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:57.503493 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:57.503521 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:57.503527 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:59:57.503548 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:57.503607 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:59:57.503627 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:57.503741 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:57.504061 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 20:59:57.504152 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210}
I20260929 20:59:57.505321 5527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:57.501147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45031 (local address 127.0.97.3:38197)
0929 20:59:57.502029 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:57.502045 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:57.502134 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:57.502687 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:57.502693 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:57.502716 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:59:57.503857 (+ 1141us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:59:57.503860 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:59:57.503876 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:59:57.503915 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:59:57.504408 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:59:57.504416 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:59:57.504419 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 20:59:57.504524 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 20:59:57.504620 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"thread_start_us":598,"threads_started":1}
I20260929 20:59:57.506114 5325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1074ac828048feac0ab69d9f7287d3" candidate_uuid: "b77bac84be7640b2b52fcf54d5460e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3c0bbf89f417c9bd7bcd3aa9ed6d9" is_pre_election: true
I20260929 20:59:57.506263 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1074ac828048feac0ab69d9f7287d3" candidate_uuid: "b77bac84be7640b2b52fcf54d5460e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80e2651f85f241ef8b701cb1e78e3f1c" is_pre_election: true
I20260929 20:59:57.506804 5325 raft_consensus.cc:2454] T 3e1074ac828048feac0ab69d9f7287d3 P a2a3c0bbf89f417c9bd7bcd3aa9ed6d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b77bac84be7640b2b52fcf54d5460e92 in term 0.
I20260929 20:59:57.506848 5194 raft_consensus.cc:2454] T 3e1074ac828048feac0ab69d9f7287d3 P 80e2651f85f241ef8b701cb1e78e3f1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b77bac84be7640b2b52fcf54d5460e92 in term 0.
I20260929 20:59:57.507575 5388 leader_election.cc:302] T 3e1074ac828048feac0ab69d9f7287d3 P b77bac84be7640b2b52fcf54d5460e92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80e2651f85f241ef8b701cb1e78e3f1c, b77bac84be7640b2b52fcf54d5460e92; no voters:
I20260929 20:59:57.508154 5524 raft_consensus.cc:2781] T 3e1074ac828048feac0ab69d9f7287d3 P b77bac84be7640b2b52fcf54d5460e92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:59:57.508311 5524 raft_consensus.cc:482] T 3e1074ac828048feac0ab69d9f7287d3 P b77bac84be7640b2b52fcf54d5460e92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:59:57.508420 5524 raft_consensus.cc:3037] T 3e1074ac828048feac0ab69d9f7287d3 P b77bac84be7640b2b52fcf54d5460e92 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:57.511267 5524 raft_consensus.cc:504] T 3e1074ac828048feac0ab69d9f7287d3 P b77bac84be7640b2b52fcf54d5460e92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a3c0bbf89f417c9bd7bcd3aa9ed6d9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45973 } } peers { permanent_uuid: "b77bac84be7640b2b52fcf54d5460e92" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37747 } } peers { permanent_uuid: "80e2651f85f241ef8b701cb1e78e3f1c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45031 } }
I20260929 20:59:57.512391 5325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1074ac828048feac0ab69d9f7287d3" candidate_uuid: "b77bac84be7640b2b52fcf54d5460e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3c0bbf89f417c9bd7bcd3aa9ed6d9"
I20260929 20:59:57.512614 5325 raft_consensus.cc:3037] T 3e1074ac828048feac0ab69d9f7287d3 P a2a3c0bbf89f417c9bd7bcd3aa9ed6d9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:57.512887 5524 leader_election.cc:288] T 3e1074ac828048feac0ab69d9f7287d3 P b77bac84be7640b2b52fcf54d5460e92 [CANDIDATE]: Term 1 election: Requested vote from peers a2a3c0bbf89f417c9bd7bcd3aa9ed6d9 (127.0.97.2:45973), 80e2651f85f241ef8b701cb1e78e3f1c (127.0.97.1:45031)
I20260929 20:59:57.512957 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1074ac828048feac0ab69d9f7287d3" candidate_uuid: "b77bac84be7640b2b52fcf54d5460e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80e2651f85f241ef8b701cb1e78e3f1c"
I20260929 20:59:57.513154 5194 raft_consensus.cc:3037] T 3e1074ac828048feac0ab69d9f7287d3 P 80e2651f85f241ef8b701cb1e78e3f1c [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:57.517268 5194 raft_consensus.cc:2454] T 3e1074ac828048feac0ab69d9f7287d3 P 80e2651f85f241ef8b701cb1e78e3f1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b77bac84be7640b2b52fcf54d5460e92 in term 1.
I20260929 20:59:57.517732 5388 leader_election.cc:302] T 3e1074ac828048feac0ab69d9f7287d3 P b77bac84be7640b2b52fcf54d5460e92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80e2651f85f241ef8b701cb1e78e3f1c, b77bac84be7640b2b52fcf54d5460e92; no voters:
I20260929 20:59:57.518018 5524 raft_consensus.cc:2781] T 3e1074ac828048feac0ab69d9f7287d3 P b77bac84be7640b2b52fcf54d5460e92 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:59:57.518128 5325 raft_consensus.cc:2454] T 3e1074ac828048feac0ab69d9f7287d3 P a2a3c0bbf89f417c9bd7bcd3aa9ed6d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b77bac84be7640b2b52fcf54d5460e92 in term 1.
I20260929 20:59:57.518498 5524 raft_consensus.cc:686] T 3e1074ac828048feac0ab69d9f7287d3 P b77bac84be7640b2b52fcf54d5460e92 [term 1 LEADER]: Becoming Leader. State: Replica: b77bac84be7640b2b52fcf54d5460e92, State: Running, Role: LEADER
I20260929 20:59:57.519019 5524 consensus_queue.cc:237] T 3e1074ac828048feac0ab69d9f7287d3 P b77bac84be7640b2b52fcf54d5460e92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a3c0bbf89f417c9bd7bcd3aa9ed6d9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45973 } } peers { permanent_uuid: "b77bac84be7640b2b52fcf54d5460e92" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37747 } } peers { permanent_uuid: "80e2651f85f241ef8b701cb1e78e3f1c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45031 } }
W20260929 20:59:57.523343 5240 tablet.cc:2369] T 3e1074ac828048feac0ab69d9f7287d3 P 80e2651f85f241ef8b701cb1e78e3f1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:59:57.524667 5053 catalog_manager.cc:5495] T 3e1074ac828048feac0ab69d9f7287d3 P b77bac84be7640b2b52fcf54d5460e92 reported cstate change: term changed from 0 to 1, leader changed from <none> to b77bac84be7640b2b52fcf54d5460e92 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "b77bac84be7640b2b52fcf54d5460e92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a3c0bbf89f417c9bd7bcd3aa9ed6d9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b77bac84be7640b2b52fcf54d5460e92" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80e2651f85f241ef8b701cb1e78e3f1c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45031 } health_report { overall_health: UNKNOWN } } }
I20260929 20:59:57.541181 5516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:57.539172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45644 (local address 127.0.97.3:37747)
0929 20:59:57.539353 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:57.539358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:57.539375 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:59:57.539540 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:57.539544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:57.539678 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:57.539764 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:57.540580 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:57.540584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:57.540587 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:59:57.540600 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:57.540627 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:59:57.540638 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:57.540739 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:57.540810 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 20:59:57.540904 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:59:57.548980 5325 raft_consensus.cc:1261] T 3e1074ac828048feac0ab69d9f7287d3 P a2a3c0bbf89f417c9bd7bcd3aa9ed6d9 [term 1 FOLLOWER]: Refusing update from remote peer b77bac84be7640b2b52fcf54d5460e92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:59:57.548985 5194 raft_consensus.cc:1261] T 3e1074ac828048feac0ab69d9f7287d3 P 80e2651f85f241ef8b701cb1e78e3f1c [term 1 FOLLOWER]: Refusing update from remote peer b77bac84be7640b2b52fcf54d5460e92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:59:57.549911 5528 consensus_queue.cc:1035] T 3e1074ac828048feac0ab69d9f7287d3 P b77bac84be7640b2b52fcf54d5460e92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3c0bbf89f417c9bd7bcd3aa9ed6d9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:59:57.550251 5524 consensus_queue.cc:1035] T 3e1074ac828048feac0ab69d9f7287d3 P b77bac84be7640b2b52fcf54d5460e92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80e2651f85f241ef8b701cb1e78e3f1c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:59:57.562772 5533 mvcc.cc:205] Tried to move back new op lower bound from 7334771087550930944 to 7334771087443484672. Current Snapshot: MvccSnapshot[applied={T|T < 7334771087550930944}]
I20260929 20:59:57.574604 5052 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:59:57.574913 5052 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:59:57.578361 5052 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9135f76961884c7aaac3f07577409849: Sending DeleteTablet for 3 replicas of tablet 3e1074ac828048feac0ab69d9f7287d3
I20260929 20:59:57.579380 5435 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e1074ac828048feac0ab69d9f7287d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:59:57 UTC) from {username='slave'} at 127.0.0.1:45630
I20260929 20:59:57.579373 5174 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e1074ac828048feac0ab69d9f7287d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:59:57 UTC) from {username='slave'} at 127.0.0.1:49094
I20260929 20:59:57.580087 5305 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e1074ac828048feac0ab69d9f7287d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:59:57 UTC) from {username='slave'} at 127.0.0.1:37412
I20260929 20:59:57.580910 5542 tablet_replica.cc:330] T 3e1074ac828048feac0ab69d9f7287d3 P 80e2651f85f241ef8b701cb1e78e3f1c: stopping tablet replica
I20260929 20:59:57.580560 5541 tablet_replica.cc:330] T 3e1074ac828048feac0ab69d9f7287d3 P b77bac84be7640b2b52fcf54d5460e92: stopping tablet replica
I20260929 20:59:57.581535 5541 raft_consensus.cc:2229] T 3e1074ac828048feac0ab69d9f7287d3 P b77bac84be7640b2b52fcf54d5460e92 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:59:57.582083 5542 raft_consensus.cc:2229] T 3e1074ac828048feac0ab69d9f7287d3 P 80e2651f85f241ef8b701cb1e78e3f1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:59:57.582284 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 5112
I20260929 20:59:57.583595 5542 raft_consensus.cc:2258] T 3e1074ac828048feac0ab69d9f7287d3 P 80e2651f85f241ef8b701cb1e78e3f1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:59:57.582938 5541 raft_consensus.cc:2258] T 3e1074ac828048feac0ab69d9f7287d3 P b77bac84be7640b2b52fcf54d5460e92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:59:57.586105 5542 ts_tablet_manager.cc:1905] T 3e1074ac828048feac0ab69d9f7287d3 P 80e2651f85f241ef8b701cb1e78e3f1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:59:57.587946 5541 ts_tablet_manager.cc:1905] T 3e1074ac828048feac0ab69d9f7287d3 P b77bac84be7640b2b52fcf54d5460e92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:59:57.589267 5543 tablet_replica.cc:330] T 3e1074ac828048feac0ab69d9f7287d3 P a2a3c0bbf89f417c9bd7bcd3aa9ed6d9: stopping tablet replica
I20260929 20:59:57.589663 5543 raft_consensus.cc:2229] T 3e1074ac828048feac0ab69d9f7287d3 P a2a3c0bbf89f417c9bd7bcd3aa9ed6d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:59:57.590197 5543 raft_consensus.cc:2258] T 3e1074ac828048feac0ab69d9f7287d3 P a2a3c0bbf89f417c9bd7bcd3aa9ed6d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:59:57.592017 5543 ts_tablet_manager.cc:1905] T 3e1074ac828048feac0ab69d9f7287d3 P a2a3c0bbf89f417c9bd7bcd3aa9ed6d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:59:57.592064 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:57.583124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49098 (local address 127.0.97.1:45031)
0929 20:59:57.583292 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:57.583299 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:57.588980 (+ 5681us) server_negotiation.cc:407] Connection header received
0929 20:59:57.589451 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:57.589456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:57.589534 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:57.589614 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:57.591285 (+ 1671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:57.591292 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:57.591296 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:59:57.591314 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:57.591351 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:59:57.591381 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:57.591475 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:57.591735 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 20:59:57.591797 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 20:59:57.593101 5234 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:59:57.598215 5541 ts_tablet_manager.cc:1918] T 3e1074ac828048feac0ab69d9f7287d3 P b77bac84be7640b2b52fcf54d5460e92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:59:57.598454 5541 log.cc:1196] T 3e1074ac828048feac0ab69d9f7287d3 P b77bac84be7640b2b52fcf54d5460e92: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-2/wal/wals/3e1074ac828048feac0ab69d9f7287d3
I20260929 20:59:57.599077 5541 ts_tablet_manager.cc:1939] T 3e1074ac828048feac0ab69d9f7287d3 P b77bac84be7640b2b52fcf54d5460e92: Deleting consensus metadata
I20260929 20:59:57.600890 5543 ts_tablet_manager.cc:1918] T 3e1074ac828048feac0ab69d9f7287d3 P a2a3c0bbf89f417c9bd7bcd3aa9ed6d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:59:57.601087 5543 log.cc:1196] T 3e1074ac828048feac0ab69d9f7287d3 P a2a3c0bbf89f417c9bd7bcd3aa9ed6d9: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790715556572573-388-0/minicluster-data/ts-1/wal/wals/3e1074ac828048feac0ab69d9f7287d3
I20260929 20:59:57.601084 5040 catalog_manager.cc:4841] TS b77bac84be7640b2b52fcf54d5460e92 (127.0.97.3:37747): tablet 3e1074ac828048feac0ab69d9f7287d3 (table test-table [id=85ae27d63d6c4941b0abbfe7301dc655]) successfully deleted
I20260929 20:59:57.601665 5543 ts_tablet_manager.cc:1939] T 3e1074ac828048feac0ab69d9f7287d3 P a2a3c0bbf89f417c9bd7bcd3aa9ed6d9: Deleting consensus metadata
I20260929 20:59:57.603726 5038 catalog_manager.cc:4841] TS a2a3c0bbf89f417c9bd7bcd3aa9ed6d9 (127.0.97.2:45973): tablet 3e1074ac828048feac0ab69d9f7287d3 (table test-table [id=85ae27d63d6c4941b0abbfe7301dc655]) successfully deleted
I20260929 20:59:57.690502 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 5112
I20260929 20:59:57.708439 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 5243
W20260929 20:59:57.708467 5037 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:45031 (error 108)
I20260929 20:59:57.711925 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:57.709493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37420 (local address 127.0.97.2:45973)
0929 20:59:57.709663 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:57.709669 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:57.709690 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:59:57.709834 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:57.709839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:57.709942 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:57.710050 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:57.711227 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:57.711234 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:57.711239 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:59:57.711257 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:57.711296 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:59:57.711311 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:57.711440 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:57.711571 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 20:59:57.711629 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:59:57.712682 5365 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:59:57.714329 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:57.713867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45031 (local address 127.0.0.1:49114)
0929 20:59:57.714050 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:57.714189 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 20:59:57.714555 5037 catalog_manager.cc:4584] TS 80e2651f85f241ef8b701cb1e78e3f1c (127.0.97.1:45031): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3e1074ac828048feac0ab69d9f7287d3: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45031: connect: Connection refused (error 111)
I20260929 20:59:57.768180 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:57.767619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45031 (local address 127.0.0.1:49128)
0929 20:59:57.767873 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:57.767987 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260929 20:59:57.769316 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:57.768943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45031 (local address 127.0.0.1:49144)
0929 20:59:57.769090 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:57.769174 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 20:59:57.800589 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 5243
I20260929 20:59:57.817483 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 5373
I20260929 20:59:57.820406 5516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:57.818256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45654 (local address 127.0.97.3:37747)
0929 20:59:57.818461 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:57.818467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:57.818489 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:59:57.818655 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:57.818661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:57.818734 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:57.818847 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:57.819722 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:57.819729 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:57.819734 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:59:57.819757 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:57.819798 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:59:57.819813 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:57.819946 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:57.820010 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 20:59:57.820063 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 20:59:57.821203 5495 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:59:57.829835 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:57.829423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45031 (local address 127.0.0.1:49158)
0929 20:59:57.829596 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:57.829690 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 20:59:57.830778 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:57.830445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45031 (local address 127.0.0.1:49166)
0929 20:59:57.830584 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:57.830663 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 20:59:57.911311 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:57.910847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45031 (local address 127.0.0.1:49170)
0929 20:59:57.911056 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:57.911147 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 20:59:57.912323 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:57.911936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45031 (local address 127.0.0.1:49180)
0929 20:59:57.912142 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:57.912205 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260929 20:59:57.937579 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 5373
I20260929 20:59:57.956326 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 5022
I20260929 20:59:57.957530 5083 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:59:58.060395 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 5022
2026-09-29T20:59:58Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2513 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 20:59:58.100497 388 mini_oidc.cc:152] Starting JWKS server
I20260929 20:59:58.103039 388 webserver.cc:460] Webserver started at https://127.0.0.1:33945/ using document root <none> and password file <none>
I20260929 20:59:58.103351 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 20:59:58.103972 388 webserver.cc:460] Webserver started at http://127.0.0.1:46205/ using document root <none> and password file <none>
2026-09-29T20:59:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:59:58Z Disabled control of system clock
I20260929 20:59:58.128160 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40475
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:36139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40475
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33945/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715556572573-388-0/testchainca.cert with env {}
W20260929 20:59:58.444514 5555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:58.444852 5555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:58.445003 5555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:58.452253 5555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:59:58.452370 5555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:58.452409 5555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:58.452461 5555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:59:58.452504 5555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 20:59:58.452558 5555 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 20:59:58.452598 5555 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33945/jwks.json
I20260929 20:59:58.462744 5555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40475
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33945/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715556572573-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:58.464097 5555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:58.465361 5555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:58.476826 5560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:58.476930 5561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:58.476881 5563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:58.482270 5555 server_base.cc:1034] running on GCE node
I20260929 20:59:58.483088 5555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:58.484508 5555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:58.485782 5555 hybrid_clock.cc:648] HybridClock initialized: now 1790715598485742 us; error 54 us; skew 500 ppm
I20260929 20:59:58.486158 5555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:59:58.488451 5555 webserver.cc:460] Webserver started at http://127.0.97.62:36441/ using document root <none> and password file <none>
I20260929 20:59:58.489207 5555 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:58.489339 5555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:58.489634 5555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:58.492436 5555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "b12841d7a9f54b32b96781f65ebb2baa"
format_stamp: "Formatted at 2026-09-29 20:59:58 on dist-test-slave-1dms"
server_key: "0f47bb78c31abcfc0b0718f629b33b5c"
server_key_iv: "75c742161c72da117758d946be459f63"
server_key_version: "encryptionkey@0"
I20260929 20:59:58.493238 5555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "b12841d7a9f54b32b96781f65ebb2baa"
format_stamp: "Formatted at 2026-09-29 20:59:58 on dist-test-slave-1dms"
server_key: "0f47bb78c31abcfc0b0718f629b33b5c"
server_key_iv: "75c742161c72da117758d946be459f63"
server_key_version: "encryptionkey@0"
I20260929 20:59:58.498098 5555 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 20:59:58.501480 5569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:58.503346 5555 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 20:59:58.503549 5555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "b12841d7a9f54b32b96781f65ebb2baa"
format_stamp: "Formatted at 2026-09-29 20:59:58 on dist-test-slave-1dms"
server_key: "0f47bb78c31abcfc0b0718f629b33b5c"
server_key_iv: "75c742161c72da117758d946be459f63"
server_key_version: "encryptionkey@0"
I20260929 20:59:58.503799 5555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:58.532855 5555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:58.533628 5555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:58.533874 5555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:58.551613 5555 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40475
I20260929 20:59:58.551633 5620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40475 every 8 connection(s)
I20260929 20:59:58.552945 5555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:59:58.554811 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 5555
I20260929 20:59:58.555016 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 20:59:58.555641 388 external_mini_cluster.cc:1428] Setting key 256d9152e93096d6212d32dc03991176
I20260929 20:59:58.556942 5621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:58.570778 5624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:58.557540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47282 (local address 127.0.97.62:40475)
0929 20:59:58.558487 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:58.558507 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:58.558546 (+ 39us) server_negotiation.cc:407] Connection header received
0929 20:59:58.559988 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:58.560008 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:58.560537 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:58.561672 (+ 1135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:58.562198 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:58.563892 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:58.565866 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:58.567051 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:58.567251 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:58.568499 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:58.568506 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:58.568518 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:59:58.569161 (+ 643us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:58.569241 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:59:58.569260 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:58.569380 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:58.569603 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 20:59:58.569897 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":345,"threads_started":1}
I20260929 20:59:58.570962 5621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b12841d7a9f54b32b96781f65ebb2baa: Bootstrap starting.
I20260929 20:59:58.574321 5621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b12841d7a9f54b32b96781f65ebb2baa: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:58.574324 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:59:58.575256 5621 log.cc:824] T 00000000000000000000000000000000 P b12841d7a9f54b32b96781f65ebb2baa: Log is configured to *not* fsync() on all Append() calls
I20260929 20:59:58.578125 5621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b12841d7a9f54b32b96781f65ebb2baa: No bootstrap required, opened a new log
I20260929 20:59:58.585299 5621 raft_consensus.cc:348] T 00000000000000000000000000000000 P b12841d7a9f54b32b96781f65ebb2baa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b12841d7a9f54b32b96781f65ebb2baa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40475 } }
I20260929 20:59:58.585661 5621 raft_consensus.cc:374] T 00000000000000000000000000000000 P b12841d7a9f54b32b96781f65ebb2baa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:58.585764 5621 raft_consensus.cc:729] T 00000000000000000000000000000000 P b12841d7a9f54b32b96781f65ebb2baa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b12841d7a9f54b32b96781f65ebb2baa, State: Initialized, Role: FOLLOWER
I20260929 20:59:58.586351 5621 consensus_queue.cc:260] T 00000000000000000000000000000000 P b12841d7a9f54b32b96781f65ebb2baa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b12841d7a9f54b32b96781f65ebb2baa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40475 } }
I20260929 20:59:58.586635 5621 raft_consensus.cc:388] T 00000000000000000000000000000000 P b12841d7a9f54b32b96781f65ebb2baa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:59:58.586804 5621 raft_consensus.cc:482] T 00000000000000000000000000000000 P b12841d7a9f54b32b96781f65ebb2baa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:59:58.586987 5621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b12841d7a9f54b32b96781f65ebb2baa [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:58.589039 5621 raft_consensus.cc:504] T 00000000000000000000000000000000 P b12841d7a9f54b32b96781f65ebb2baa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b12841d7a9f54b32b96781f65ebb2baa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40475 } }
I20260929 20:59:58.589463 5621 leader_election.cc:302] T 00000000000000000000000000000000 P b12841d7a9f54b32b96781f65ebb2baa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b12841d7a9f54b32b96781f65ebb2baa; no voters:
I20260929 20:59:58.589972 5621 leader_election.cc:288] T 00000000000000000000000000000000 P b12841d7a9f54b32b96781f65ebb2baa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:59:58.590130 5626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b12841d7a9f54b32b96781f65ebb2baa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:59:58.590783 5626 raft_consensus.cc:686] T 00000000000000000000000000000000 P b12841d7a9f54b32b96781f65ebb2baa [term 1 LEADER]: Becoming Leader. State: Replica: b12841d7a9f54b32b96781f65ebb2baa, State: Running, Role: LEADER
I20260929 20:59:58.591298 5626 consensus_queue.cc:237] T 00000000000000000000000000000000 P b12841d7a9f54b32b96781f65ebb2baa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b12841d7a9f54b32b96781f65ebb2baa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40475 } }
I20260929 20:59:58.591954 5621 sys_catalog.cc:564] T 00000000000000000000000000000000 P b12841d7a9f54b32b96781f65ebb2baa [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:59:58.594053 5628 sys_catalog.cc:455] T 00000000000000000000000000000000 P b12841d7a9f54b32b96781f65ebb2baa [sys.catalog]: SysCatalogTable state changed. Reason: New leader b12841d7a9f54b32b96781f65ebb2baa. Latest consensus state: current_term: 1 leader_uuid: "b12841d7a9f54b32b96781f65ebb2baa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b12841d7a9f54b32b96781f65ebb2baa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40475 } } }
I20260929 20:59:58.594415 5627 sys_catalog.cc:455] T 00000000000000000000000000000000 P b12841d7a9f54b32b96781f65ebb2baa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b12841d7a9f54b32b96781f65ebb2baa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b12841d7a9f54b32b96781f65ebb2baa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40475 } } }
I20260929 20:59:58.595095 5627 sys_catalog.cc:458] T 00000000000000000000000000000000 P b12841d7a9f54b32b96781f65ebb2baa [sys.catalog]: This master's current role is: LEADER
I20260929 20:59:58.594421 5628 sys_catalog.cc:458] T 00000000000000000000000000000000 P b12841d7a9f54b32b96781f65ebb2baa [sys.catalog]: This master's current role is: LEADER
I20260929 20:59:58.596091 5635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:59:58.600667 5635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:59:58.608311 5635 catalog_manager.cc:1347] Generated new cluster ID: 4e7e16cb51784bbe9e821fac309cdaa0
I20260929 20:59:58.608407 5635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:59:58.638860 5635 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:59:58.640677 5635 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:59:58.664318 5635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b12841d7a9f54b32b96781f65ebb2baa: Generated new TSK 0
I20260929 20:59:58.665419 5635 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 20:59:58.702966 5651 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 20:59:58.703137 5651 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 20:59:58.704150 5624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:58.683104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47294 (local address 127.0.97.62:40475)
0929 20:59:58.683339 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:58.683345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:58.683756 (+ 411us) server_negotiation.cc:407] Connection header received
0929 20:59:58.684054 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:58.684057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:58.684126 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:58.684248 (+ 122us) server_negotiation.cc:226] Negotiated authn=JWT
0929 20:59:58.684937 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:58.685667 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:58.687731 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:58.688177 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:58.688277 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:58.688923 (+ 646us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 20:59:58.703676 (+ 14753us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33945/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 20:59:58.703934 (+ 258us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47294: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33945/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 20:59:58.704432 5624 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47294: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33945/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 20:59:58.706199 5650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:59:58.682988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40475 (local address 127.0.0.1:47294)
0929 20:59:58.683635 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:58.683665 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:58.683843 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:58.684297 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:58.684302 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:58.684315 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0929 20:59:58.684735 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:58.684749 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:58.685808 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:58.685814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:58.687576 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:58.687590 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:58.688293 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:58.688354 (+ 61us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:58.688538 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:58.688608 (+ 70us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 20:59:58.703941 (+ 15333us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33945/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 20:59:58.704289 (+ 348us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:40475: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33945/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":328,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:40475: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33945/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 20:59:58.708397 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 5555
I20260929 20:59:58.709486 5616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:59:58.806942 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 5555
2026-09-29T20:59:58Z chronyd exiting
I20260929 20:59:58.857318 388 test_util.cc:175] -----------------------------------------------
I20260929 20:59:58.857450 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790715556572573-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (761 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 20:59:58.860978 388 mini_oidc.cc:152] Starting JWKS server
I20260929 20:59:58.862704 388 webserver.cc:460] Webserver started at https://127.0.0.1:35581/ using document root <none> and password file <none>
I20260929 20:59:58.862891 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 20:59:58.863481 388 webserver.cc:460] Webserver started at http://127.0.0.1:34977/ using document root <none> and password file <none>
2026-09-29T20:59:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:59:58Z Disabled control of system clock
I20260929 20:59:58.889012 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41435
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41435
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35581/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790715556572573-388-0/testchainca.cert with env {}
W20260929 20:59:59.197152 5658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:59.197472 5658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:59.197638 5658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:59.204707 5658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:59:59.204845 5658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:59.204957 5658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:59.205045 5658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:59:59.205111 5658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 20:59:59.205195 5658 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 20:59:59.205278 5658 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35581/jwks.json
I20260929 20:59:59.214917 5658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41435
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35581/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790715556572573-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:59.216167 5658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:59.217458 5658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:59.228693 5663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:59.228727 5664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:59.228883 5666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:59.229423 5658 server_base.cc:1034] running on GCE node
I20260929 20:59:59.230122 5658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:59.231465 5658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:59.232681 5658 hybrid_clock.cc:648] HybridClock initialized: now 1790715599232632 us; error 45 us; skew 500 ppm
I20260929 20:59:59.233067 5658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:59:59.235241 5658 webserver.cc:460] Webserver started at http://127.0.97.62:33881/ using document root <none> and password file <none>
I20260929 20:59:59.235899 5658 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:59.236030 5658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:59.236351 5658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:59.240818 5658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "ff3fdbe29af44d709b260b32696a7eed"
format_stamp: "Formatted at 2026-09-29 20:59:59 on dist-test-slave-1dms"
server_key: "99c09ef564a4cdae307f375a520dad93"
server_key_iv: "195de8d04005e1a8a1b969343b6002dc"
server_key_version: "encryptionkey@0"
I20260929 20:59:59.241654 5658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "ff3fdbe29af44d709b260b32696a7eed"
format_stamp: "Formatted at 2026-09-29 20:59:59 on dist-test-slave-1dms"
server_key: "99c09ef564a4cdae307f375a520dad93"
server_key_iv: "195de8d04005e1a8a1b969343b6002dc"
server_key_version: "encryptionkey@0"
I20260929 20:59:59.246340 5658 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 20:59:59.249570 5672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:59.250806 5658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:59:59.250982 5658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "ff3fdbe29af44d709b260b32696a7eed"
format_stamp: "Formatted at 2026-09-29 20:59:59 on dist-test-slave-1dms"
server_key: "99c09ef564a4cdae307f375a520dad93"
server_key_iv: "195de8d04005e1a8a1b969343b6002dc"
server_key_version: "encryptionkey@0"
I20260929 20:59:59.251183 5658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:59.278908 5658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:59.279784 5658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:59.280073 5658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:59:59.298025 5658 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41435
I20260929 20:59:59.298044 5723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41435 every 8 connection(s)
I20260929 20:59:59.299366 5658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:59:59.303076 5724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:59:59.305248 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 5658
I20260929 20:59:59.305446 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 20:59:59.306072 388 external_mini_cluster.cc:1428] Setting key b3eab4df4e8ee7841a551d70782787b9
I20260929 20:59:59.314623 5724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff3fdbe29af44d709b260b32696a7eed: Bootstrap starting.
I20260929 20:59:59.318169 5724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff3fdbe29af44d709b260b32696a7eed: Neither blocks nor log segments found. Creating new log.
I20260929 20:59:59.319298 5724 log.cc:824] T 00000000000000000000000000000000 P ff3fdbe29af44d709b260b32696a7eed: Log is configured to *not* fsync() on all Append() calls
I20260929 20:59:59.319947 5727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:59.308050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48814 (local address 127.0.97.62:41435)
0929 20:59:59.309148 (+ 1098us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:59.309162 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:59.309203 (+ 41us) server_negotiation.cc:407] Connection header received
0929 20:59:59.310801 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:59.310824 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:59.311293 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:59.311627 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:59:59.312208 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:59.313578 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:59.315325 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:59.316049 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:59.316203 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:59.317511 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:59:59.317519 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:59:59.317532 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:59:59.318048 (+ 516us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:59:59.318143 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:59:59.318168 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:59:59.318285 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 20:59:59.318522 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 20:59:59.318888 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":382,"threads_started":1}
I20260929 20:59:59.322510 5724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff3fdbe29af44d709b260b32696a7eed: No bootstrap required, opened a new log
I20260929 20:59:59.323796 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:59:59.330755 5724 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff3fdbe29af44d709b260b32696a7eed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff3fdbe29af44d709b260b32696a7eed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41435 } }
I20260929 20:59:59.331094 5724 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff3fdbe29af44d709b260b32696a7eed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:59:59.331211 5724 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff3fdbe29af44d709b260b32696a7eed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff3fdbe29af44d709b260b32696a7eed, State: Initialized, Role: FOLLOWER
I20260929 20:59:59.331738 5724 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff3fdbe29af44d709b260b32696a7eed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff3fdbe29af44d709b260b32696a7eed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41435 } }
I20260929 20:59:59.331955 5724 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff3fdbe29af44d709b260b32696a7eed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:59:59.332100 5724 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff3fdbe29af44d709b260b32696a7eed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:59:59.332259 5724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff3fdbe29af44d709b260b32696a7eed [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:59:59.334455 5724 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff3fdbe29af44d709b260b32696a7eed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff3fdbe29af44d709b260b32696a7eed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41435 } }
I20260929 20:59:59.334923 5724 leader_election.cc:302] T 00000000000000000000000000000000 P ff3fdbe29af44d709b260b32696a7eed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff3fdbe29af44d709b260b32696a7eed; no voters:
I20260929 20:59:59.335507 5724 leader_election.cc:288] T 00000000000000000000000000000000 P ff3fdbe29af44d709b260b32696a7eed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:59:59.335613 5729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff3fdbe29af44d709b260b32696a7eed [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:59:59.336814 5729 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff3fdbe29af44d709b260b32696a7eed [term 1 LEADER]: Becoming Leader. State: Replica: ff3fdbe29af44d709b260b32696a7eed, State: Running, Role: LEADER
I20260929 20:59:59.337321 5729 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff3fdbe29af44d709b260b32696a7eed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff3fdbe29af44d709b260b32696a7eed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41435 } }
I20260929 20:59:59.337689 5724 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff3fdbe29af44d709b260b32696a7eed [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:59:59.340087 5731 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff3fdbe29af44d709b260b32696a7eed [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff3fdbe29af44d709b260b32696a7eed. Latest consensus state: current_term: 1 leader_uuid: "ff3fdbe29af44d709b260b32696a7eed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff3fdbe29af44d709b260b32696a7eed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41435 } } }
I20260929 20:59:59.340605 5731 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff3fdbe29af44d709b260b32696a7eed [sys.catalog]: This master's current role is: LEADER
I20260929 20:59:59.340075 5730 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff3fdbe29af44d709b260b32696a7eed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff3fdbe29af44d709b260b32696a7eed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff3fdbe29af44d709b260b32696a7eed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41435 } } }
I20260929 20:59:59.341028 5730 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff3fdbe29af44d709b260b32696a7eed [sys.catalog]: This master's current role is: LEADER
I20260929 20:59:59.345117 5736 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:59:59.350028 5736 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:59:59.358822 5736 catalog_manager.cc:1347] Generated new cluster ID: 8f58e79bb8be4987be7579168ddbeb25
I20260929 20:59:59.358923 5736 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:59:59.381376 5736 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:59:59.383126 5736 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:59:59.393898 5736 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff3fdbe29af44d709b260b32696a7eed: Generated new TSK 0
I20260929 20:59:59.394738 5736 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 20:59:59.433348 5754 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 20:59:59.433475 5754 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 20:59:59.434736 5727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:59:59.414889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48824 (local address 127.0.97.62:41435)
0929 20:59:59.415080 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 20:59:59.415087 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:59:59.415532 (+ 445us) server_negotiation.cc:407] Connection header received
0929 20:59:59.415753 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:59:59.415759 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:59:59.415853 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:59:59.416009 (+ 156us) server_negotiation.cc:226] Negotiated authn=JWT
0929 20:59:59.416465 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:59.417587 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:59.418906 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:59:59.419579 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:59:59.419708 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:59.420125 (+ 417us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 20:59:59.434043 (+ 13918us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35581/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 20:59:59.434438 (+ 395us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48824: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35581/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":66}
W20260929 20:59:59.434840 5753 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:59:59.414706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41435 (local address 127.0.0.1:48824)
0929 20:59:59.415435 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0929 20:59:59.415451 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:59:59.415576 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:59:59.416030 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:59:59.416034 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:59:59.416043 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0929 20:59:59.416311 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:59.416319 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:59.417718 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:59.417722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:59.418775 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:59:59.418785 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:59:59.419690 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:59:59.419726 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:59:59.419849 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:59:59.419878 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 20:59:59.434328 (+ 14450us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35581/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 20:59:59.434512 (+ 184us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:41435: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35581/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":421,"threads_started":1}
W20260929 20:59:59.434870 5727 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48824: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35581/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260929 20:59:59.436502 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 5658
I20260929 20:59:59.437794 5719 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:59:59.529115 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 5658
2026-09-29T20:59:59Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (724 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 20:59:59.586247 388 mini_oidc.cc:152] Starting JWKS server
I20260929 20:59:59.588140 388 webserver.cc:460] Webserver started at https://127.0.0.1:40455/ using document root <none> and password file <none>
I20260929 20:59:59.588415 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 20:59:59.589089 388 webserver.cc:460] Webserver started at http://127.0.0.1:45585/ using document root <none> and password file <none>
2026-09-29T20:59:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:59:59Z Disabled control of system clock
I20260929 20:59:59.615224 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37159
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:44061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37159
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40455/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 20:59:59.922669 5761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:59:59.923002 5761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:59:59.923168 5761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:59:59.930199 5761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:59:59.930339 5761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:59:59.930411 5761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:59:59.930495 5761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:59:59.930579 5761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 20:59:59.930665 5761 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 20:59:59.930749 5761 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40455/jwks.json
I20260929 20:59:59.939596 5761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37159
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40455/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:59:59.940873 5761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:59:59.942617 5761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:59:59.953606 5767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:59.953568 5766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:59:59.953873 5769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:59:59.954483 5761 server_base.cc:1034] running on GCE node
I20260929 20:59:59.955181 5761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:59:59.956494 5761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:59:59.957697 5761 hybrid_clock.cc:648] HybridClock initialized: now 1790715599957659 us; error 27 us; skew 500 ppm
I20260929 20:59:59.958062 5761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:59:59.960186 5761 webserver.cc:460] Webserver started at http://127.0.97.62:42891/ using document root <none> and password file <none>
I20260929 20:59:59.960909 5761 fs_manager.cc:359] Metadata directory not provided
I20260929 20:59:59.961015 5761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:59:59.961347 5761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:59:59.964437 5761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "a20795bfed1b4c5ba8d0475984b70b80"
format_stamp: "Formatted at 2026-09-29 20:59:59 on dist-test-slave-1dms"
server_key: "43f1b5b1812d7d56f13a5d7339a0607b"
server_key_iv: "1b7cd77786465f1adec333ca1a4cf1eb"
server_key_version: "encryptionkey@0"
I20260929 20:59:59.965445 5761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "a20795bfed1b4c5ba8d0475984b70b80"
format_stamp: "Formatted at 2026-09-29 20:59:59 on dist-test-slave-1dms"
server_key: "43f1b5b1812d7d56f13a5d7339a0607b"
server_key_iv: "1b7cd77786465f1adec333ca1a4cf1eb"
server_key_version: "encryptionkey@0"
I20260929 20:59:59.970330 5761 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:59:59.973621 5775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:59:59.974910 5761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 20:59:59.975085 5761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "a20795bfed1b4c5ba8d0475984b70b80"
format_stamp: "Formatted at 2026-09-29 20:59:59 on dist-test-slave-1dms"
server_key: "43f1b5b1812d7d56f13a5d7339a0607b"
server_key_iv: "1b7cd77786465f1adec333ca1a4cf1eb"
server_key_version: "encryptionkey@0"
I20260929 20:59:59.975287 5761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:59:59.995935 5761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:59:59.996722 5761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:59:59.997020 5761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:00.015085 5761 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37159
I20260929 21:00:00.015107 5826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37159 every 8 connection(s)
I20260929 21:00:00.016422 5761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:00:00.020100 5827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:00:00.021675 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 5761
I20260929 21:00:00.021888 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:00:00.022704 388 external_mini_cluster.cc:1428] Setting key 69db9f9bab07577cdb107759138a4a51
I20260929 21:00:00.033841 5827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a20795bfed1b4c5ba8d0475984b70b80: Bootstrap starting.
I20260929 21:00:00.035701 5830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:00.024501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50772 (local address 127.0.97.62:37159)
0929 21:00:00.025489 (+ 988us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:00.025499 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:00.025538 (+ 39us) server_negotiation.cc:407] Connection header received
0929 21:00:00.027031 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:00.027051 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:00.027497 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:00.027797 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:00.028354 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:00.029635 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:00.031461 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:00.032123 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:00.032245 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:00.033536 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:00.033544 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:00.033556 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:00:00.034105 (+ 549us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:00.034179 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:00:00.034198 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:00.034286 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:00.034495 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 21:00:00.034808 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":379,"threads_started":1}
I20260929 21:00:00.037053 5827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a20795bfed1b4c5ba8d0475984b70b80: Neither blocks nor log segments found. Creating new log.
I20260929 21:00:00.038100 5827 log.cc:824] T 00000000000000000000000000000000 P a20795bfed1b4c5ba8d0475984b70b80: Log is configured to *not* fsync() on all Append() calls
I20260929 21:00:00.039263 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 21:00:00.040745 5827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a20795bfed1b4c5ba8d0475984b70b80: No bootstrap required, opened a new log
I20260929 21:00:00.048246 5827 raft_consensus.cc:348] T 00000000000000000000000000000000 P a20795bfed1b4c5ba8d0475984b70b80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20795bfed1b4c5ba8d0475984b70b80" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37159 } }
I20260929 21:00:00.048586 5827 raft_consensus.cc:374] T 00000000000000000000000000000000 P a20795bfed1b4c5ba8d0475984b70b80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:00:00.048689 5827 raft_consensus.cc:729] T 00000000000000000000000000000000 P a20795bfed1b4c5ba8d0475984b70b80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20795bfed1b4c5ba8d0475984b70b80, State: Initialized, Role: FOLLOWER
I20260929 21:00:00.049232 5827 consensus_queue.cc:260] T 00000000000000000000000000000000 P a20795bfed1b4c5ba8d0475984b70b80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20795bfed1b4c5ba8d0475984b70b80" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37159 } }
I20260929 21:00:00.049470 5827 raft_consensus.cc:388] T 00000000000000000000000000000000 P a20795bfed1b4c5ba8d0475984b70b80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:00:00.049604 5827 raft_consensus.cc:482] T 00000000000000000000000000000000 P a20795bfed1b4c5ba8d0475984b70b80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:00:00.049790 5827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a20795bfed1b4c5ba8d0475984b70b80 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:00:00.052682 5827 raft_consensus.cc:504] T 00000000000000000000000000000000 P a20795bfed1b4c5ba8d0475984b70b80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20795bfed1b4c5ba8d0475984b70b80" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37159 } }
I20260929 21:00:00.053181 5827 leader_election.cc:302] T 00000000000000000000000000000000 P a20795bfed1b4c5ba8d0475984b70b80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a20795bfed1b4c5ba8d0475984b70b80; no voters:
I20260929 21:00:00.053712 5827 leader_election.cc:288] T 00000000000000000000000000000000 P a20795bfed1b4c5ba8d0475984b70b80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 21:00:00.053871 5832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a20795bfed1b4c5ba8d0475984b70b80 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:00:00.054504 5832 raft_consensus.cc:686] T 00000000000000000000000000000000 P a20795bfed1b4c5ba8d0475984b70b80 [term 1 LEADER]: Becoming Leader. State: Replica: a20795bfed1b4c5ba8d0475984b70b80, State: Running, Role: LEADER
I20260929 21:00:00.054967 5832 consensus_queue.cc:237] T 00000000000000000000000000000000 P a20795bfed1b4c5ba8d0475984b70b80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20795bfed1b4c5ba8d0475984b70b80" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37159 } }
I20260929 21:00:00.055653 5827 sys_catalog.cc:564] T 00000000000000000000000000000000 P a20795bfed1b4c5ba8d0475984b70b80 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:00:00.057825 5833 sys_catalog.cc:455] T 00000000000000000000000000000000 P a20795bfed1b4c5ba8d0475984b70b80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a20795bfed1b4c5ba8d0475984b70b80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20795bfed1b4c5ba8d0475984b70b80" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37159 } } }
I20260929 21:00:00.058252 5833 sys_catalog.cc:458] T 00000000000000000000000000000000 P a20795bfed1b4c5ba8d0475984b70b80 [sys.catalog]: This master's current role is: LEADER
I20260929 21:00:00.058698 5834 sys_catalog.cc:455] T 00000000000000000000000000000000 P a20795bfed1b4c5ba8d0475984b70b80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a20795bfed1b4c5ba8d0475984b70b80. Latest consensus state: current_term: 1 leader_uuid: "a20795bfed1b4c5ba8d0475984b70b80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20795bfed1b4c5ba8d0475984b70b80" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37159 } } }
I20260929 21:00:00.059001 5834 sys_catalog.cc:458] T 00000000000000000000000000000000 P a20795bfed1b4c5ba8d0475984b70b80 [sys.catalog]: This master's current role is: LEADER
I20260929 21:00:00.060112 5838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:00:00.065068 5838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:00:00.073781 5838 catalog_manager.cc:1347] Generated new cluster ID: f6034e0db34a4bbf9e00ae9129fae26e
I20260929 21:00:00.073925 5838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:00:00.089865 5838 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:00:00.091634 5838 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:00:00.114251 5838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a20795bfed1b4c5ba8d0475984b70b80: Generated new TSK 0
I20260929 21:00:00.115324 5838 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 21:00:00.149657 5830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:00.141165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50776 (local address 127.0.97.62:37159)
0929 21:00:00.141362 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:00.141369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:00.141795 (+ 426us) server_negotiation.cc:407] Connection header received
0929 21:00:00.142001 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:00.142007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:00.142099 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:00.142281 (+ 182us) server_negotiation.cc:226] Negotiated authn=JWT
0929 21:00:00.142705 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:00.143837 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:00.145248 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:00.145961 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:00.146087 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:00.146557 (+ 470us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 21:00:00.149018 (+ 2461us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40455/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 21:00:00.149352 (+ 334us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50776: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40455/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 21:00:00.149785 5830 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50776: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40455/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 21:00:00.151520 5856 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:00.141056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37159 (local address 127.0.0.1:50776)
0929 21:00:00.141688 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:00.141707 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:00.141831 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:00.142285 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:00.142288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:00.142298 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0929 21:00:00.142565 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:00.142573 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:00.143991 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:00.143995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:00.145114 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:00.145126 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:00.146078 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:00.146137 (+ 59us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:00.146266 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:00.146296 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 21:00:00.149206 (+ 2910us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40455/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 21:00:00.151112 (+ 1906us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:37159: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40455/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":1616,"thread_start_us":342,"threads_started":1}
I20260929 21:00:00.152691 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 5761
I20260929 21:00:00.153834 5822 generic_service.cc:196] Checking for leaks (request via RPC)
E20260929 21:00:00.162853 5857 webserver.cc:533] Webserver: sslize failed:
E20260929 21:00:00.162936 5857 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 21:00:00.252743 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 5761
2026-09-29T21:00:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (722 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T21:00:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:00:00Z Disabled control of system clock
I20260929 21:00:00.333678 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46203
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:41275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46203
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790715556572573-388-0/testchainca.cert with env {}
W20260929 21:00:00.649032 5862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:00.649312 5862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:00.649397 5862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:00.656306 5862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:00:00.656411 5862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:00.656450 5862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:00.656505 5862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:00:00.656550 5862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 21:00:00.656603 5862 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 21:00:00.656642 5862 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 21:00:00.666181 5862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46203
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790715556572573-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:00:00.667397 5862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:00.668615 5862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:00.679786 5867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:00.679813 5868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:00.680022 5870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:00.680667 5862 server_base.cc:1034] running on GCE node
I20260929 21:00:00.681413 5862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:00.682751 5862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:00.683939 5862 hybrid_clock.cc:648] HybridClock initialized: now 1790715600683902 us; error 26 us; skew 500 ppm
I20260929 21:00:00.684278 5862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 21:00:00.686465 5862 webserver.cc:460] Webserver started at http://127.0.97.62:36519/ using document root <none> and password file <none>
I20260929 21:00:00.687110 5862 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:00.687232 5862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:00.687543 5862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:00:00.690347 5862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "ee0e42fbecb44361a727de9c9915fde0"
format_stamp: "Formatted at 2026-09-29 21:00:00 on dist-test-slave-1dms"
server_key: "0f61444ad48c9e029586f88b090ce3a3"
server_key_iv: "cb39e3db7531d48d15fdb76ccdd7ec97"
server_key_version: "encryptionkey@0"
I20260929 21:00:00.691112 5862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "ee0e42fbecb44361a727de9c9915fde0"
format_stamp: "Formatted at 2026-09-29 21:00:00 on dist-test-slave-1dms"
server_key: "0f61444ad48c9e029586f88b090ce3a3"
server_key_iv: "cb39e3db7531d48d15fdb76ccdd7ec97"
server_key_version: "encryptionkey@0"
I20260929 21:00:00.695823 5862 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 21:00:00.699121 5876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:00.700451 5862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 21:00:00.700644 5862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "ee0e42fbecb44361a727de9c9915fde0"
format_stamp: "Formatted at 2026-09-29 21:00:00 on dist-test-slave-1dms"
server_key: "0f61444ad48c9e029586f88b090ce3a3"
server_key_iv: "cb39e3db7531d48d15fdb76ccdd7ec97"
server_key_version: "encryptionkey@0"
I20260929 21:00:00.700876 5862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:00.721477 5862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:00.722252 5862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:00:00.722544 5862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:00.740065 5862 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46203
I20260929 21:00:00.740087 5927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46203 every 8 connection(s)
I20260929 21:00:00.741348 5862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:00:00.744814 5928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:00:00.749967 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 5862
I20260929 21:00:00.750144 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:00:00.750942 388 external_mini_cluster.cc:1428] Setting key 254b6e60fea6b428bfacd2a12326c989
I20260929 21:00:00.755810 5928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee0e42fbecb44361a727de9c9915fde0: Bootstrap starting.
I20260929 21:00:00.758877 5928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee0e42fbecb44361a727de9c9915fde0: Neither blocks nor log segments found. Creating new log.
I20260929 21:00:00.759842 5928 log.cc:824] T 00000000000000000000000000000000 P ee0e42fbecb44361a727de9c9915fde0: Log is configured to *not* fsync() on all Append() calls
I20260929 21:00:00.762557 5928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee0e42fbecb44361a727de9c9915fde0: No bootstrap required, opened a new log
I20260929 21:00:00.763818 5931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:00.752945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52186 (local address 127.0.97.62:46203)
0929 21:00:00.753934 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:00.753945 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:00.753988 (+ 43us) server_negotiation.cc:407] Connection header received
0929 21:00:00.755572 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:00.755595 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:00.756084 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:00.756453 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:00.757014 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:00.758279 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:00.759581 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:00.760331 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:00.760557 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:00.761671 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:00.761679 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:00.761692 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:00:00.762201 (+ 509us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:00.762275 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:00:00.762295 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:00.762428 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:00.762668 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 21:00:00.762946 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":339,"threads_started":1}
I20260929 21:00:00.767863 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 21:00:00.771365 5928 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee0e42fbecb44361a727de9c9915fde0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee0e42fbecb44361a727de9c9915fde0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46203 } }
I20260929 21:00:00.771726 5928 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee0e42fbecb44361a727de9c9915fde0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:00:00.771847 5928 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee0e42fbecb44361a727de9c9915fde0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0e42fbecb44361a727de9c9915fde0, State: Initialized, Role: FOLLOWER
I20260929 21:00:00.772486 5928 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee0e42fbecb44361a727de9c9915fde0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee0e42fbecb44361a727de9c9915fde0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46203 } }
I20260929 21:00:00.772716 5928 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee0e42fbecb44361a727de9c9915fde0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:00:00.772881 5928 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee0e42fbecb44361a727de9c9915fde0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:00:00.773058 5928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee0e42fbecb44361a727de9c9915fde0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:00:00.775570 5928 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee0e42fbecb44361a727de9c9915fde0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee0e42fbecb44361a727de9c9915fde0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46203 } }
I20260929 21:00:00.776055 5928 leader_election.cc:302] T 00000000000000000000000000000000 P ee0e42fbecb44361a727de9c9915fde0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee0e42fbecb44361a727de9c9915fde0; no voters:
I20260929 21:00:00.776644 5928 leader_election.cc:288] T 00000000000000000000000000000000 P ee0e42fbecb44361a727de9c9915fde0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 21:00:00.776767 5933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee0e42fbecb44361a727de9c9915fde0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:00:00.777491 5933 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee0e42fbecb44361a727de9c9915fde0 [term 1 LEADER]: Becoming Leader. State: Replica: ee0e42fbecb44361a727de9c9915fde0, State: Running, Role: LEADER
I20260929 21:00:00.778083 5933 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee0e42fbecb44361a727de9c9915fde0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee0e42fbecb44361a727de9c9915fde0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46203 } }
I20260929 21:00:00.778820 5928 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee0e42fbecb44361a727de9c9915fde0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:00:00.781160 5935 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee0e42fbecb44361a727de9c9915fde0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee0e42fbecb44361a727de9c9915fde0. Latest consensus state: current_term: 1 leader_uuid: "ee0e42fbecb44361a727de9c9915fde0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee0e42fbecb44361a727de9c9915fde0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46203 } } }
I20260929 21:00:00.781574 5935 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee0e42fbecb44361a727de9c9915fde0 [sys.catalog]: This master's current role is: LEADER
I20260929 21:00:00.782116 5934 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee0e42fbecb44361a727de9c9915fde0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee0e42fbecb44361a727de9c9915fde0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee0e42fbecb44361a727de9c9915fde0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46203 } } }
I20260929 21:00:00.782482 5934 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee0e42fbecb44361a727de9c9915fde0 [sys.catalog]: This master's current role is: LEADER
I20260929 21:00:00.783960 5942 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:00:00.788851 5942 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:00:00.797577 5942 catalog_manager.cc:1347] Generated new cluster ID: d6968b654c884283809e662fb8ed2478
I20260929 21:00:00.797740 5942 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:00:00.820890 5942 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:00:00.822171 5942 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:00:00.826925 5942 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee0e42fbecb44361a727de9c9915fde0: Generated new TSK 0
I20260929 21:00:00.827673 5942 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 21:00:00.921006 5931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:00.854457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52190 (local address 127.0.97.62:46203)
0929 21:00:00.854653 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:00.854660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:00.855056 (+ 396us) server_negotiation.cc:407] Connection header received
0929 21:00:00.855250 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:00.855257 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:00.855350 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:00.855509 (+ 159us) server_negotiation.cc:226] Negotiated authn=JWT
0929 21:00:00.856002 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:00.857177 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:00.858556 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:00.859256 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:00.859381 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:00.859851 (+ 470us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 21:00:00.920225 (+ 60374us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 21:00:00.920642 (+ 417us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52190: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 21:00:00.921145 5931 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52190: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 21:00:00.921156 5957 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:00.854311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46203 (local address 127.0.0.1:52190)
0929 21:00:00.854958 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:00.854974 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:00.855088 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:00.855568 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:00.855572 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:00.855589 (+ 17us) client_negotiation.cc:189] Negotiated authn=JWT
0929 21:00:00.855850 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:00.855858 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:00.857330 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:00.857334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:00.858410 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:00.858430 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:00.859383 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:00.859427 (+ 44us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:00.859565 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:00.859598 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 21:00:00.920525 (+ 60927us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 21:00:00.920745 (+ 220us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:46203: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":380,"threads_started":1}
I20260929 21:00:00.922875 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 5862
I20260929 21:00:00.924027 5923 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:00:01.021674 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 5862
2026-09-29T21:00:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (747 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (402 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (401 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 21:00:01 dist-test-slave-1dms krb5kdc[5972](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 21:00:01 dist-test-slave-1dms krb5kdc[5972](info): set up 2 sockets
Sep 29 21:00:01 dist-test-slave-1dms krb5kdc[5972](info): commencing operation
krb5kdc: starting...
W20260929 21:00:02.929562 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.004s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 21:00:03 dist-test-slave-1dms krb5kdc[5972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T21:00:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:00:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 21:00:03.125887 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41643
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41643
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:00:03.441238 5985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:03.441567 5985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:03.441742 5985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:03.448808 5985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:00:03.448984 5985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:03.449092 5985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:03.449198 5985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:00:03.449286 5985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:00:03.458921 5985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:00:03.460273 5985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:03.461738 5985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:03.473348 5990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:03.473308 5991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:03.473372 5993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:03.474392 5985 server_base.cc:1034] running on GCE node
I20260929 21:00:03.475046 5985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:03.476264 5985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:03.477452 5985 hybrid_clock.cc:648] HybridClock initialized: now 1790715603477413 us; error 33 us; skew 500 ppm
I20260929 21:00:03.477756 5985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:00:03 dist-test-slave-1dms krb5kdc[5972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:00:03.480782 5985 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 21:00:03.482158 5985 webserver.cc:460] Webserver started at http://127.0.97.62:41879/ using document root <none> and password file <none>
I20260929 21:00:03.482762 5985 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:03.482856 5985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:03.483103 5985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:00:03.485977 5985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "b83ca5e4b378402f8956891943e75865"
format_stamp: "Formatted at 2026-09-29 21:00:03 on dist-test-slave-1dms"
server_key: "50dcb32ecccf3468ef09fed916ff85a2"
server_key_iv: "74a65737724bee7c659bbb9238a4dfba"
server_key_version: "encryptionkey@0"
I20260929 21:00:03.486646 5985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "b83ca5e4b378402f8956891943e75865"
format_stamp: "Formatted at 2026-09-29 21:00:03 on dist-test-slave-1dms"
server_key: "50dcb32ecccf3468ef09fed916ff85a2"
server_key_iv: "74a65737724bee7c659bbb9238a4dfba"
server_key_version: "encryptionkey@0"
I20260929 21:00:03.491272 5985 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 21:00:03.494550 6000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:03.495814 5985 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 21:00:03.495982 5985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "b83ca5e4b378402f8956891943e75865"
format_stamp: "Formatted at 2026-09-29 21:00:03 on dist-test-slave-1dms"
server_key: "50dcb32ecccf3468ef09fed916ff85a2"
server_key_iv: "74a65737724bee7c659bbb9238a4dfba"
server_key_version: "encryptionkey@0"
I20260929 21:00:03.496160 5985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:03.532097 5985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:03.536069 5985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:00:03.536350 5985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:03.554013 5985 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41643
I20260929 21:00:03.554090 6051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41643 every 8 connection(s)
I20260929 21:00:03.555305 5985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:00:03.558789 6052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:00:03.565259 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 5985
I20260929 21:00:03.565459 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:00:03.566074 388 external_mini_cluster.cc:1428] Setting key 7af69904e6e51e42c523d4f33cd5af88
I20260929 21:00:03.568763 6052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b83ca5e4b378402f8956891943e75865: Bootstrap starting.
I20260929 21:00:03.572019 6052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b83ca5e4b378402f8956891943e75865: Neither blocks nor log segments found. Creating new log.
I20260929 21:00:03.573242 6052 log.cc:824] T 00000000000000000000000000000000 P b83ca5e4b378402f8956891943e75865: Log is configured to *not* fsync() on all Append() calls
I20260929 21:00:03.576287 6052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b83ca5e4b378402f8956891943e75865: No bootstrap required, opened a new log
Sep 29 21:00:03 dist-test-slave-1dms krb5kdc[5972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:00:03.586673 6052 raft_consensus.cc:348] T 00000000000000000000000000000000 P b83ca5e4b378402f8956891943e75865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b83ca5e4b378402f8956891943e75865" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41643 } }
I20260929 21:00:03.587109 6052 raft_consensus.cc:374] T 00000000000000000000000000000000 P b83ca5e4b378402f8956891943e75865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:00:03.587211 6052 raft_consensus.cc:729] T 00000000000000000000000000000000 P b83ca5e4b378402f8956891943e75865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b83ca5e4b378402f8956891943e75865, State: Initialized, Role: FOLLOWER
I20260929 21:00:03.587863 6052 consensus_queue.cc:260] T 00000000000000000000000000000000 P b83ca5e4b378402f8956891943e75865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b83ca5e4b378402f8956891943e75865" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41643 } }
I20260929 21:00:03.588162 6052 raft_consensus.cc:388] T 00000000000000000000000000000000 P b83ca5e4b378402f8956891943e75865 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:00:03.588299 6052 raft_consensus.cc:482] T 00000000000000000000000000000000 P b83ca5e4b378402f8956891943e75865 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:00:03.588466 6052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b83ca5e4b378402f8956891943e75865 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:00:03.589968 6055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:03.567872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53512 (local address 127.0.97.62:41643)
0929 21:00:03.568589 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:03.568599 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:03.568637 (+ 38us) server_negotiation.cc:407] Connection header received
0929 21:00:03.570389 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:03.570409 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:03.570770 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:03.571058 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:03.572789 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:03.574551 (+ 1762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:03.576229 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:03.576969 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:03.577100 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:03.581591 (+ 4491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:03.581600 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:03.581623 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:03.581670 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:03.585536 (+ 3866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:03.586172 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:03.586181 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:03.586189 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:03.586407 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:03.586835 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:03.586839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:03.586842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:03.587524 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:03.587793 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:03.588168 (+ 375us) server_negotiation.cc:299] Negotiation successful
0929 21:00:03.588601 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":341,"threads_started":1}
I20260929 21:00:03.591250 6052 raft_consensus.cc:504] T 00000000000000000000000000000000 P b83ca5e4b378402f8956891943e75865 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b83ca5e4b378402f8956891943e75865" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41643 } }
I20260929 21:00:03.591728 6052 leader_election.cc:302] T 00000000000000000000000000000000 P b83ca5e4b378402f8956891943e75865 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b83ca5e4b378402f8956891943e75865; no voters:
I20260929 21:00:03.592427 6052 leader_election.cc:288] T 00000000000000000000000000000000 P b83ca5e4b378402f8956891943e75865 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 21:00:03.592527 6057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b83ca5e4b378402f8956891943e75865 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:00:03.593132 6057 raft_consensus.cc:686] T 00000000000000000000000000000000 P b83ca5e4b378402f8956891943e75865 [term 1 LEADER]: Becoming Leader. State: Replica: b83ca5e4b378402f8956891943e75865, State: Running, Role: LEADER
I20260929 21:00:03.593623 6057 consensus_queue.cc:237] T 00000000000000000000000000000000 P b83ca5e4b378402f8956891943e75865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b83ca5e4b378402f8956891943e75865" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41643 } }
I20260929 21:00:03.594558 6052 sys_catalog.cc:564] T 00000000000000000000000000000000 P b83ca5e4b378402f8956891943e75865 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:00:03.596578 6059 sys_catalog.cc:455] T 00000000000000000000000000000000 P b83ca5e4b378402f8956891943e75865 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b83ca5e4b378402f8956891943e75865. Latest consensus state: current_term: 1 leader_uuid: "b83ca5e4b378402f8956891943e75865" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b83ca5e4b378402f8956891943e75865" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41643 } } }
I20260929 21:00:03.596556 6058 sys_catalog.cc:455] T 00000000000000000000000000000000 P b83ca5e4b378402f8956891943e75865 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b83ca5e4b378402f8956891943e75865" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b83ca5e4b378402f8956891943e75865" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41643 } } }
I20260929 21:00:03.597023 6059 sys_catalog.cc:458] T 00000000000000000000000000000000 P b83ca5e4b378402f8956891943e75865 [sys.catalog]: This master's current role is: LEADER
I20260929 21:00:03.597060 6058 sys_catalog.cc:458] T 00000000000000000000000000000000 P b83ca5e4b378402f8956891943e75865 [sys.catalog]: This master's current role is: LEADER
I20260929 21:00:03.598160 6062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:00:03.603961 6062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:00:03.612602 6062 catalog_manager.cc:1347] Generated new cluster ID: e1c886e39bd7428ba0f7ca3a11007935
I20260929 21:00:03.612701 6062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:00:03.625613 6062 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:00:03.626907 6062 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:00:03.635090 6062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b83ca5e4b378402f8956891943e75865: Generated new TSK 0
I20260929 21:00:03.636036 6062 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 21:00:03.688719 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41643
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:00:04.003993 6078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:04.004287 6078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:04.004477 6078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:04.011679 6078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:04.011837 6078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:04.012009 6078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 21:00:04.021553 6078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:00:04.022807 6078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:04.024058 6078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:04.036547 6084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:04.036573 6083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:04.036661 6086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:04.037397 6078 server_base.cc:1034] running on GCE node
I20260929 21:00:04.037976 6078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:04.038847 6078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:04.040041 6078 hybrid_clock.cc:648] HybridClock initialized: now 1790715604039999 us; error 33 us; skew 500 ppm
I20260929 21:00:04.040367 6078 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:00:04 dist-test-slave-1dms krb5kdc[5972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:00:04.043438 6078 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 21:00:04.044895 6078 webserver.cc:460] Webserver started at http://127.0.97.1:36831/ using document root <none> and password file <none>
I20260929 21:00:04.045553 6078 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:04.045722 6078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:04.046015 6078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:00:04.048801 6078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-0/data/instance:
uuid: "6cca456c4b8b4f4ea2ead5eec5a05280"
format_stamp: "Formatted at 2026-09-29 21:00:04 on dist-test-slave-1dms"
server_key: "42a861bd75d09078d289a299c978472c"
server_key_iv: "6fbf2f7d0399e92aabd7c5e21da1e2db"
server_key_version: "encryptionkey@0"
I20260929 21:00:04.049649 6078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6cca456c4b8b4f4ea2ead5eec5a05280"
format_stamp: "Formatted at 2026-09-29 21:00:04 on dist-test-slave-1dms"
server_key: "42a861bd75d09078d289a299c978472c"
server_key_iv: "6fbf2f7d0399e92aabd7c5e21da1e2db"
server_key_version: "encryptionkey@0"
I20260929 21:00:04.054366 6078 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 21:00:04.057577 6093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:04.058890 6078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:00:04.059057 6078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-0/wal
uuid: "6cca456c4b8b4f4ea2ead5eec5a05280"
format_stamp: "Formatted at 2026-09-29 21:00:04 on dist-test-slave-1dms"
server_key: "42a861bd75d09078d289a299c978472c"
server_key_iv: "6fbf2f7d0399e92aabd7c5e21da1e2db"
server_key_version: "encryptionkey@0"
I20260929 21:00:04.059253 6078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:04.079779 6078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:04.083681 6078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:00:04.083968 6078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:04.084991 6078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:00:04.086261 6078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:00:04.086364 6078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:04.086505 6078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:00:04.086576 6078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:04.119561 6078 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43793
I20260929 21:00:04.119616 6205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43793 every 8 connection(s)
I20260929 21:00:04.120923 6078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
I20260929 21:00:04.128403 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 6078
I20260929 21:00:04.128592 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-0/wal/instance
I20260929 21:00:04.129276 388 external_mini_cluster.cc:1428] Setting key 68824b975ffaba52f8a388b3e3526d06
Sep 29 21:00:04 dist-test-slave-1dms krb5kdc[5972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:00:04.141216 6208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:04.123378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55195 (local address 127.0.97.62:41643)
0929 21:00:04.123816 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:04.123822 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:04.124862 (+ 1040us) server_negotiation.cc:407] Connection header received
0929 21:00:04.126629 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:04.126633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:04.126695 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:04.126795 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:04.128356 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:04.129246 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:04.131152 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:04.131517 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:04.131609 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:04.135686 (+ 4077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:04.135691 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:04.135694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:04.135722 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:04.138309 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:04.139013 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:04.139025 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:04.139028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:04.139136 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:04.139547 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:04.139554 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:04.139558 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:04.139819 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:04.139969 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:04.140600 (+ 631us) server_negotiation.cc:299] Negotiation successful
0929 21:00:04.140820 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":206,"threads_started":1}
I20260929 21:00:04.142164 6209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:04.123985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41643 (local address 127.0.97.1:55195)
0929 21:00:04.124670 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:04.124712 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:04.126371 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:04.126994 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:04.127002 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:04.127510 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:04.128182 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:04.128207 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:04.129397 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:04.129403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:04.131001 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:04.131012 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:04.131728 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:04.131734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:04.132066 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:04.132787 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:04.132820 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:04.135476 (+ 2656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:04.138447 (+ 2971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:04.138455 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:04.138472 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:04.138886 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:04.139253 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:04.139258 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:04.139262 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:04.139438 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:04.139943 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:04.139951 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:04.140270 (+ 319us) client_negotiation.cc:769] Sending connection context
0929 21:00:04.140529 (+ 259us) client_negotiation.cc:240] Negotiation successful
0929 21:00:04.141080 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":284,"threads_started":1}
I20260929 21:00:04.143685 6206 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41643
I20260929 21:00:04.144011 6206 heartbeater.cc:464] Registering TS with master...
I20260929 21:00:04.144941 6206 heartbeater.cc:511] Master 127.0.97.62:41643 requested a full tablet report, sending...
I20260929 21:00:04.147387 6017 ts_manager.cc:194] Registered new tserver with Master: 6cca456c4b8b4f4ea2ead5eec5a05280 (127.0.97.1:43793)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 21:00:04.149600 6017 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55195
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 21:00:04.176086 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41643
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:00:04.489988 6212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:04.490253 6212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:04.490347 6212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:04.497488 6212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:04.497589 6212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:04.497710 6212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 21:00:04.508469 6212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:00:04.509827 6212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:04.511046 6212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:04.523633 6220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:04.523587 6218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:04.523726 6217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:04.524703 6212 server_base.cc:1034] running on GCE node
I20260929 21:00:04.525305 6212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:04.526185 6212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:04.527386 6212 hybrid_clock.cc:648] HybridClock initialized: now 1790715604527349 us; error 31 us; skew 500 ppm
I20260929 21:00:04.527685 6212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:00:04 dist-test-slave-1dms krb5kdc[5972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:00:04.530721 6212 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 21:00:04.532022 6212 webserver.cc:460] Webserver started at http://127.0.97.2:36071/ using document root <none> and password file <none>
I20260929 21:00:04.532662 6212 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:04.532801 6212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:04.533109 6212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:00:04.535873 6212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-1/data/instance:
uuid: "6f1f744227f94062a103d6061eebbded"
format_stamp: "Formatted at 2026-09-29 21:00:04 on dist-test-slave-1dms"
server_key: "43d8eadaad5a68d14054858378e238e6"
server_key_iv: "34c73726350f8000eca77b3c0710374f"
server_key_version: "encryptionkey@0"
I20260929 21:00:04.536617 6212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6f1f744227f94062a103d6061eebbded"
format_stamp: "Formatted at 2026-09-29 21:00:04 on dist-test-slave-1dms"
server_key: "43d8eadaad5a68d14054858378e238e6"
server_key_iv: "34c73726350f8000eca77b3c0710374f"
server_key_version: "encryptionkey@0"
I20260929 21:00:04.541558 6212 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260929 21:00:04.545228 6227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:04.546514 6212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 21:00:04.546635 6212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-1/wal
uuid: "6f1f744227f94062a103d6061eebbded"
format_stamp: "Formatted at 2026-09-29 21:00:04 on dist-test-slave-1dms"
server_key: "43d8eadaad5a68d14054858378e238e6"
server_key_iv: "34c73726350f8000eca77b3c0710374f"
server_key_version: "encryptionkey@0"
I20260929 21:00:04.546805 6212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:04.577402 6212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:04.581283 6212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:00:04.581516 6212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:04.582381 6212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:00:04.583621 6212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:00:04.583690 6212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:04.583772 6212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:00:04.583815 6212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:04.616395 6212 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44971
I20260929 21:00:04.616439 6339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44971 every 8 connection(s)
I20260929 21:00:04.617689 6212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
I20260929 21:00:04.624349 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 6212
I20260929 21:00:04.624531 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-1/wal/instance
I20260929 21:00:04.625030 388 external_mini_cluster.cc:1428] Setting key 69f2c0f0877042fb6a7eafa952c812cc
Sep 29 21:00:04 dist-test-slave-1dms krb5kdc[5972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:00:04.638851 6208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:04.620220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59447 (local address 127.0.97.62:41643)
0929 21:00:04.620384 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:04.620388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:04.621559 (+ 1171us) server_negotiation.cc:407] Connection header received
0929 21:00:04.623444 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:04.623449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:04.623544 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:04.623656 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:04.625316 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:04.626117 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:04.627751 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:04.628550 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:04.628692 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:04.632004 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:04.632009 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:04.632012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:04.632041 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:04.635398 (+ 3357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:04.636302 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:04.636311 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:04.636317 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:04.636466 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:04.636894 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:04.636902 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:04.636907 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:04.637267 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:04.637412 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:04.638067 (+ 655us) server_negotiation.cc:299] Negotiation successful
0929 21:00:04.638307 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 21:00:04.639461 6342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:04.620627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41643 (local address 127.0.97.2:59447)
0929 21:00:04.621395 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:04.621438 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:04.623166 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:04.623900 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:04.623908 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:04.624411 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:04.625137 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:04.625161 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:04.626280 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:04.626284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:04.627605 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:04.627616 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:04.628756 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:04.628767 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:04.629019 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:04.629698 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:04.629720 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:04.631797 (+ 2077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:04.635537 (+ 3740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:04.635544 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:04.635556 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:04.636142 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:04.636571 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:04.636575 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:04.636577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:04.636757 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:04.637383 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:04.637389 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:04.637694 (+ 305us) client_negotiation.cc:769] Sending connection context
0929 21:00:04.637916 (+ 222us) client_negotiation.cc:240] Negotiation successful
0929 21:00:04.638291 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":360,"threads_started":1}
I20260929 21:00:04.640991 6340 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41643
I20260929 21:00:04.641311 6340 heartbeater.cc:464] Registering TS with master...
I20260929 21:00:04.642225 6340 heartbeater.cc:511] Master 127.0.97.62:41643 requested a full tablet report, sending...
I20260929 21:00:04.643894 6017 ts_manager.cc:194] Registered new tserver with Master: 6f1f744227f94062a103d6061eebbded (127.0.97.2:44971)
I20260929 21:00:04.644667 6017 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59447
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 21:00:04.671252 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41643
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:00:04.982882 6345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:04.983155 6345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:04.983318 6345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:04.990434 6345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:04.990535 6345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:04.990715 6345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 21:00:05.000490 6345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:00:05.001776 6345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:05.003082 6345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:05.015935 6351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:05.015962 6353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:05.016013 6350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:05.016741 6345 server_base.cc:1034] running on GCE node
I20260929 21:00:05.017400 6345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:05.018308 6345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:05.019528 6345 hybrid_clock.cc:648] HybridClock initialized: now 1790715605019491 us; error 64 us; skew 500 ppm
I20260929 21:00:05.019858 6345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:00:05 dist-test-slave-1dms krb5kdc[5972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:00:05.022979 6345 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 21:00:05.024310 6345 webserver.cc:460] Webserver started at http://127.0.97.3:45913/ using document root <none> and password file <none>
I20260929 21:00:05.025043 6345 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:05.025185 6345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:05.025483 6345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:00:05.028296 6345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-2/data/instance:
uuid: "6098431fc0d34eeea3d87fc05971b836"
format_stamp: "Formatted at 2026-09-29 21:00:05 on dist-test-slave-1dms"
server_key: "da07d6adc0a87d02d2a22b41df648a32"
server_key_iv: "8c06bce5540a7b13901762881a335d4b"
server_key_version: "encryptionkey@0"
I20260929 21:00:05.029105 6345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6098431fc0d34eeea3d87fc05971b836"
format_stamp: "Formatted at 2026-09-29 21:00:05 on dist-test-slave-1dms"
server_key: "da07d6adc0a87d02d2a22b41df648a32"
server_key_iv: "8c06bce5540a7b13901762881a335d4b"
server_key_version: "encryptionkey@0"
I20260929 21:00:05.033991 6345 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 21:00:05.037220 6360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:05.038477 6345 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 21:00:05.038646 6345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-2/wal
uuid: "6098431fc0d34eeea3d87fc05971b836"
format_stamp: "Formatted at 2026-09-29 21:00:05 on dist-test-slave-1dms"
server_key: "da07d6adc0a87d02d2a22b41df648a32"
server_key_iv: "8c06bce5540a7b13901762881a335d4b"
server_key_version: "encryptionkey@0"
I20260929 21:00:05.038841 6345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:05.056221 6345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:05.059948 6345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:00:05.060227 6345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:05.061169 6345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:00:05.062495 6345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:00:05.062563 6345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:05.062707 6345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:00:05.062803 6345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:05.094551 6345 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36489
I20260929 21:00:05.094637 6472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36489 every 8 connection(s)
I20260929 21:00:05.095968 6345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
I20260929 21:00:05.098789 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 6345
I20260929 21:00:05.098948 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/minicluster-data/ts-2/wal/instance
I20260929 21:00:05.099385 388 external_mini_cluster.cc:1428] Setting key f02dfc87ea825728f888016bf54ea018
Sep 29 21:00:05 dist-test-slave-1dms krb5kdc[5972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:00:05.115617 6208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:05.098385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46389 (local address 127.0.97.62:41643)
0929 21:00:05.098595 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:05.098603 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:05.099985 (+ 1382us) server_negotiation.cc:407] Connection header received
0929 21:00:05.102190 (+ 2205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:05.102195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:05.102276 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:05.102393 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:05.103945 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:05.104777 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:05.106375 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:05.106777 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:05.106909 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:05.110295 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:05.110301 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:05.110307 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:05.110344 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:05.112642 (+ 2298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:05.113451 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:05.113458 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:05.113464 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:05.113586 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:05.113996 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:05.114001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:05.114004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:05.114266 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:05.114416 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:05.115072 (+ 656us) server_negotiation.cc:299] Negotiation successful
0929 21:00:05.115274 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 21:00:05.116536 6475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:05.098911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41643 (local address 127.0.97.3:46389)
0929 21:00:05.099815 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:05.099866 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:05.101905 (+ 2039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:05.102630 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:05.102638 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:05.103209 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:05.103777 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:05.103794 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:05.104950 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:05.104956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:05.106195 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:05.106206 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:05.106999 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:05.107007 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:05.107253 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:05.107916 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:05.107941 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:05.110091 (+ 2150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:05.112796 (+ 2705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:05.112805 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:05.112821 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:05.113327 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:05.113694 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:05.113700 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:05.113704 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:05.113878 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:05.114391 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:05.114399 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:05.114719 (+ 320us) client_negotiation.cc:769] Sending connection context
0929 21:00:05.114981 (+ 262us) client_negotiation.cc:240] Negotiation successful
0929 21:00:05.115483 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":363,"threads_started":1}
I20260929 21:00:05.117798 6473 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41643
I20260929 21:00:05.118094 6473 heartbeater.cc:464] Registering TS with master...
I20260929 21:00:05.118854 6473 heartbeater.cc:511] Master 127.0.97.62:41643 requested a full tablet report, sending...
I20260929 21:00:05.120370 6017 ts_manager.cc:194] Registered new tserver with Master: 6098431fc0d34eeea3d87fc05971b836 (127.0.97.3:36489)
I20260929 21:00:05.121302 6017 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46389
I20260929 21:00:05.126704 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/krb5kdc/test-admin.keytab.
Sep 29 21:00:05 dist-test-slave-1dms krb5kdc[5972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:00:05.152479 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
I20260929 21:00:05.152606 6206 heartbeater.cc:503] Master 127.0.97.62:41643 was elected leader, sending a full tablet report...
W20260929 21:00:05.157824 6480 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:05.155404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52242 (local address 127.0.97.1:43793)
0929 21:00:05.156070 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:05.156081 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:05.156110 (+ 29us) server_negotiation.cc:407] Connection header received
0929 21:00:05.156171 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:05.156178 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:05.156394 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:05.156573 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:05.157483 (+ 910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52242: BlockingRecv error: recv got EOF from 127.0.0.1:52242 (error 108)
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":329,"threads_started":1}
W20260929 21:00:05.158170 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/krb5kdc/krb5cc' of type FILE
Sep 29 21:00:05 dist-test-slave-1dms krb5kdc[5972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:00:05.159804 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 21:00:05 dist-test-slave-1dms krb5kdc[5972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:00:05.161180 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 21:00:05 dist-test-slave-1dms krb5kdc[5972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 21:00:05.176529 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:05.162677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52252 (local address 127.0.97.1:43793)
0929 21:00:05.162887 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:05.162894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:05.163257 (+ 363us) server_negotiation.cc:407] Connection header received
0929 21:00:05.163456 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:05.163460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:05.163517 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:05.163618 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:05.164590 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:05.165600 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:05.166827 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:05.167392 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:05.167532 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:05.171112 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:05.171130 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:05.171144 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:05.171174 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:05.173851 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:05.174503 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:05.174510 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:05.174512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:05.174631 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:05.175005 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:05.175012 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:05.175015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:05.175365 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:05.175656 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:05.175990 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 21:00:05.176245 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260929 21:00:05.182238 6480 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:05.180386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52264 (local address 127.0.97.1:43793)
0929 21:00:05.180531 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:05.180535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:05.180967 (+ 432us) server_negotiation.cc:407] Connection header received
0929 21:00:05.181137 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:05.181141 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:05.181195 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:05.181279 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:05.181941 (+ 662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52264: BlockingRecv error: recv got EOF from 127.0.0.1:52264 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 21:00:05.182401 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/krb5kdc/krb5cc' of type FILE
Sep 29 21:00:05 dist-test-slave-1dms krb5kdc[5972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:00:05.183874 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 21:00:05 dist-test-slave-1dms krb5kdc[5972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:00:05.185333 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 21:00:05 dist-test-slave-1dms krb5kdc[5972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 21:00:05.199909 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:05.186607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52266 (local address 127.0.97.1:43793)
0929 21:00:05.186814 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:05.186821 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:05.187029 (+ 208us) server_negotiation.cc:407] Connection header received
0929 21:00:05.187214 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:05.187224 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:05.187275 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:05.187372 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:05.188283 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:05.189020 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:05.190205 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:05.190629 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:05.190766 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:05.194145 (+ 3379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:05.194153 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:05.194156 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:05.194197 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:05.197315 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:05.197915 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:05.197921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:05.197925 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:05.198041 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:05.198492 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:05.198497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:05.198500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:05.198821 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:05.199066 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:05.199367 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 21:00:05.199612 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260929 21:00:05.204493 6480 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:05.202737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52278 (local address 127.0.97.1:43793)
0929 21:00:05.202920 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:05.202927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:05.203215 (+ 288us) server_negotiation.cc:407] Connection header received
0929 21:00:05.203390 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:05.203395 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:05.203469 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:05.203567 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:05.204221 (+ 654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52278: BlockingRecv error: recv got EOF from 127.0.0.1:52278 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 21:00:05.204602 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790715556572573-388-0/krb5kdc/krb5cc' of type FILE
Sep 29 21:00:05 dist-test-slave-1dms krb5kdc[5972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:00:05.206136 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 21:00:05 dist-test-slave-1dms krb5kdc[5972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:00:05.207402 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 21:00:05 dist-test-slave-1dms krb5kdc[5972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 21:00:05.220628 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:05.208728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52286 (local address 127.0.97.1:43793)
0929 21:00:05.208958 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:05.208963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:05.209117 (+ 154us) server_negotiation.cc:407] Connection header received
0929 21:00:05.209232 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:05.209235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:05.209288 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:05.209363 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:05.210550 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:05.211238 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:05.212395 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:05.212776 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:05.212927 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:05.216060 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:05.216065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:05.216067 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:05.216094 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:05.218573 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:05.219102 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:05.219106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:05.219108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:05.219237 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:05.219520 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:05.219523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:05.219524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:05.219740 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:05.219962 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:05.220136 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 21:00:05.220274 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 21:00:05.222215 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 6078
I20260929 21:00:05.233126 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:05.222870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52288 (local address 127.0.97.1:43793)
0929 21:00:05.223046 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:05.223055 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:05.223325 (+ 270us) server_negotiation.cc:407] Connection header received
0929 21:00:05.223413 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:05.223417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:05.223464 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:05.223534 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:05.224393 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:05.225117 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:05.226214 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:05.226593 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:05.226679 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:05.228438 (+ 1759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:05.228443 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:05.228446 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:05.228473 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:05.230968 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:05.231519 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:05.231523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:05.231525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:05.231650 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:05.231933 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:05.231936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:05.231938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:05.232149 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:05.232322 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:05.232565 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 21:00:05.232783 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 21:00:05.233675 6201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:00:05.315951 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 6078
I20260929 21:00:05.331962 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 6212
Sep 29 21:00:05 dist-test-slave-1dms krb5kdc[5972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 21:00:05.347105 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:05.332823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56796 (local address 127.0.97.2:44971)
0929 21:00:05.333455 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:05.333462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:05.333487 (+ 25us) server_negotiation.cc:407] Connection header received
0929 21:00:05.333546 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:05.333553 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:05.333769 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:05.333895 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:05.334824 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:05.335811 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:05.337245 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:05.337784 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:05.337882 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:05.341547 (+ 3665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:05.341557 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:05.341572 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:05.341603 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:05.344337 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:05.345059 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:05.345066 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:05.345068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:05.345187 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:05.345575 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:05.345584 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:05.345586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:05.345929 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:05.346228 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:05.346554 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 21:00:05.346804 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":338,"threads_started":1}
I20260929 21:00:05.348347 6335 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:00:05.431859 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 6212
I20260929 21:00:05.447324 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 6345
Sep 29 21:00:05 dist-test-slave-1dms krb5kdc[5972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 21:00:05.462496 6495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:05.448404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42592 (local address 127.0.97.3:36489)
0929 21:00:05.449130 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:05.449136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:05.449166 (+ 30us) server_negotiation.cc:407] Connection header received
0929 21:00:05.449225 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:05.449232 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:05.449439 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:05.449570 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:05.450490 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:05.451475 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:05.452623 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:05.453270 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:05.453376 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:05.457057 (+ 3681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:05.457067 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:05.457081 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:05.457111 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:05.459892 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:05.460466 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:05.460472 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:05.460475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:05.460588 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:05.461042 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:05.461045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:05.461047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:05.461361 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:05.461578 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:05.461910 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 21:00:05.462196 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":359,"threads_started":1}
I20260929 21:00:05.463784 6468 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:00:05.546334 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 6345
I20260929 21:00:05.562014 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 5985
I20260929 21:00:05.563094 6047 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:00:05.664923 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 5985
2026-09-29T21:00:05Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3845 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 21:00:05 dist-test-slave-1dms krb5kdc[6500](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 21:00:05 dist-test-slave-1dms krb5kdc[6500](info): set up 2 sockets
Sep 29 21:00:05 dist-test-slave-1dms krb5kdc[6500](info): commencing operation
krb5kdc: starting...
W20260929 21:00:06.749586 388 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.004s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 21:00:06 dist-test-slave-1dms krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T21:00:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:00:06Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260929 21:00:06.946635 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38557
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38557
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:00:07.255230 6513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:07.255503 6513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:07.255666 6513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:07.262627 6513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:00:07.262725 6513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:07.262820 6513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:07.262924 6513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:00:07.263005 6513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:00:07.272413 6513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:00:07.273715 6513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:07.275022 6513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:07.286502 6519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:07.286537 6521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:07.286474 6518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:07.287968 6513 server_base.cc:1034] running on GCE node
I20260929 21:00:07.288743 6513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:07.290184 6513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:07.291410 6513 hybrid_clock.cc:648] HybridClock initialized: now 1790715607291360 us; error 43 us; skew 500 ppm
I20260929 21:00:07.291739 6513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:00:07 dist-test-slave-1dms krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715607, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:00:07.305711 6513 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260929 21:00:07.307307 6513 webserver.cc:460] Webserver started at http://127.0.97.62:45553/ using document root <none> and password file <none>
I20260929 21:00:07.308017 6513 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:07.308161 6513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:07.308463 6513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:00:07.311424 6513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "4b9bcd97294f4ef68830d77516fb1754"
format_stamp: "Formatted at 2026-09-29 21:00:07 on dist-test-slave-1dms"
server_key: "0e27e8b256d5c691bb799211ee1ccad6"
server_key_iv: "bca2b715bfc4c377722484338ca66f9b"
server_key_version: "encryptionkey@0"
I20260929 21:00:07.312173 6513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "4b9bcd97294f4ef68830d77516fb1754"
format_stamp: "Formatted at 2026-09-29 21:00:07 on dist-test-slave-1dms"
server_key: "0e27e8b256d5c691bb799211ee1ccad6"
server_key_iv: "bca2b715bfc4c377722484338ca66f9b"
server_key_version: "encryptionkey@0"
I20260929 21:00:07.317413 6513 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 21:00:07.320912 6528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:07.322398 6513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 21:00:07.322598 6513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "4b9bcd97294f4ef68830d77516fb1754"
format_stamp: "Formatted at 2026-09-29 21:00:07 on dist-test-slave-1dms"
server_key: "0e27e8b256d5c691bb799211ee1ccad6"
server_key_iv: "bca2b715bfc4c377722484338ca66f9b"
server_key_version: "encryptionkey@0"
I20260929 21:00:07.322806 6513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:07.347309 6513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:07.351641 6513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:00:07.351933 6513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:07.370381 6513 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38557
I20260929 21:00:07.370419 6579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38557 every 8 connection(s)
I20260929 21:00:07.371675 6513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:00:07.374346 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 6513
I20260929 21:00:07.374562 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:00:07.375212 388 external_mini_cluster.cc:1428] Setting key 240dc2987cffecbb9153b83bc436e0fc
I20260929 21:00:07.375411 6580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:00:07.389531 6580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754: Bootstrap starting.
Sep 29 21:00:07 dist-test-slave-1dms krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 21:00:07.393486 6580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754: Neither blocks nor log segments found. Creating new log.
I20260929 21:00:07.394809 6580 log.cc:824] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754: Log is configured to *not* fsync() on all Append() calls
I20260929 21:00:07.398331 6580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754: No bootstrap required, opened a new log
I20260929 21:00:07.399613 6583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:07.377148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55660 (local address 127.0.97.62:38557)
0929 21:00:07.377915 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:07.377925 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:07.377966 (+ 41us) server_negotiation.cc:407] Connection header received
0929 21:00:07.379833 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:07.379854 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:07.380230 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:07.380568 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:07.382272 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:07.383550 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:07.385381 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:07.386083 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:07.386227 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:07.391020 (+ 4793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:07.391034 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:07.391054 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:07.391120 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:07.395034 (+ 3914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:07.395890 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:07.395897 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:07.395904 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:07.396073 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:07.396505 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:07.396509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:07.396512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:07.397187 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:07.397511 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:07.397833 (+ 322us) server_negotiation.cc:299] Negotiation successful
0929 21:00:07.398257 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":360,"threads_started":1}
I20260929 21:00:07.406795 6580 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } }
I20260929 21:00:07.407130 6580 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:00:07.407198 6580 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b9bcd97294f4ef68830d77516fb1754, State: Initialized, Role: FOLLOWER
I20260929 21:00:07.407662 6580 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } }
I20260929 21:00:07.407855 6580 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:00:07.407950 6580 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:00:07.408069 6580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:00:07.410179 6580 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } }
I20260929 21:00:07.410594 6580 leader_election.cc:302] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b9bcd97294f4ef68830d77516fb1754; no voters:
I20260929 21:00:07.411167 6580 leader_election.cc:288] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 21:00:07.411316 6585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:00:07.413126 6585 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [term 1 LEADER]: Becoming Leader. State: Replica: 4b9bcd97294f4ef68830d77516fb1754, State: Running, Role: LEADER
I20260929 21:00:07.413246 6580 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:00:07.413643 6585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } }
I20260929 21:00:07.416797 6586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b9bcd97294f4ef68830d77516fb1754" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } }
I20260929 21:00:07.416817 6587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b9bcd97294f4ef68830d77516fb1754. Latest consensus state: current_term: 1 leader_uuid: "4b9bcd97294f4ef68830d77516fb1754" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } }
I20260929 21:00:07.417192 6587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [sys.catalog]: This master's current role is: LEADER
I20260929 21:00:07.417189 6586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [sys.catalog]: This master's current role is: LEADER
I20260929 21:00:07.425079 6594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:00:07.431315 6594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:00:07.432035 6602 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 21:00:07.439602 6594 catalog_manager.cc:1347] Generated new cluster ID: 1d78b66f93774de588144898a42be9a5
I20260929 21:00:07.439747 6594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:00:07.451645 6594 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:00:07.453490 6594 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:00:07.461493 6594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754: Generated new TSK 0
I20260929 21:00:07.462466 6594 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260929 21:00:07.524226 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38557
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:00:07.856489 6606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:07.856789 6606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:07.856967 6606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:07.863940 6606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:07.864085 6606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:07.864269 6606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 21:00:07.874315 6606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:00:07.875600 6606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:07.876947 6606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:07.889765 6611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:07.889832 6614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:07.889801 6612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:07.890606 6606 server_base.cc:1034] running on GCE node
I20260929 21:00:07.891145 6606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:07.891993 6606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:07.893201 6606 hybrid_clock.cc:648] HybridClock initialized: now 1790715607893164 us; error 30 us; skew 500 ppm
I20260929 21:00:07.893478 6606 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:00:07 dist-test-slave-1dms krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715607, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:00:07.896680 6606 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260929 21:00:07.898046 6606 webserver.cc:460] Webserver started at http://127.0.97.1:40345/ using document root <none> and password file <none>
I20260929 21:00:07.898742 6606 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:07.898887 6606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:07.899174 6606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:00:07.902161 6606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-0/data/instance:
uuid: "527d3ead14f6443b9fef7865f31d7227"
format_stamp: "Formatted at 2026-09-29 21:00:07 on dist-test-slave-1dms"
server_key: "17e956282f8d0e3b5f64512941eea73a"
server_key_iv: "3ea27eac97cee66086f6a302bc7087e7"
server_key_version: "encryptionkey@0"
I20260929 21:00:07.902921 6606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-0/wal/instance:
uuid: "527d3ead14f6443b9fef7865f31d7227"
format_stamp: "Formatted at 2026-09-29 21:00:07 on dist-test-slave-1dms"
server_key: "17e956282f8d0e3b5f64512941eea73a"
server_key_iv: "3ea27eac97cee66086f6a302bc7087e7"
server_key_version: "encryptionkey@0"
I20260929 21:00:07.908042 6606 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 21:00:07.911378 6621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:07.912648 6606 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 21:00:07.912814 6606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-0/wal
uuid: "527d3ead14f6443b9fef7865f31d7227"
format_stamp: "Formatted at 2026-09-29 21:00:07 on dist-test-slave-1dms"
server_key: "17e956282f8d0e3b5f64512941eea73a"
server_key_iv: "3ea27eac97cee66086f6a302bc7087e7"
server_key_version: "encryptionkey@0"
I20260929 21:00:07.913056 6606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:07.927155 6606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:07.930945 6606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:00:07.931224 6606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:07.932116 6606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:00:07.933400 6606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:00:07.933499 6606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:07.933638 6606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:00:07.933712 6606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:07.966189 6606 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41637
I20260929 21:00:07.966245 6733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41637 every 8 connection(s)
I20260929 21:00:07.967475 6606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
I20260929 21:00:07.975407 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 6606
I20260929 21:00:07.975620 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-0/wal/instance
I20260929 21:00:07.976241 388 external_mini_cluster.cc:1428] Setting key 3dc37c0205a72411754e7b036bc48d10
Sep 29 21:00:07 dist-test-slave-1dms krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715607, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 21:00:07.987522 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:07.969874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60907 (local address 127.0.97.62:38557)
0929 21:00:07.970308 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:07.970313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:07.971331 (+ 1018us) server_negotiation.cc:407] Connection header received
0929 21:00:07.973083 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:07.973087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:07.973150 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:07.973249 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:07.974638 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:07.975320 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:07.977058 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:07.977595 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:07.977712 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:07.981137 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:07.981146 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:07.981153 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:07.981195 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:07.984390 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:07.985353 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:07.985357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:07.985359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:07.985465 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:07.985945 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:07.985948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:07.985949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:07.986189 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:07.986293 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:07.987123 (+ 830us) server_negotiation.cc:299] Negotiation successful
0929 21:00:07.987275 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":307,"thread_start_us":198,"threads_started":1}
I20260929 21:00:07.989029 6737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:07.970500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38557 (local address 127.0.97.1:60907)
0929 21:00:07.971178 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:07.971214 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:07.972783 (+ 1569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:07.973440 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:07.973448 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:07.973967 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:07.974496 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:07.974512 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:07.975475 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:07.975485 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:07.976906 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:07.976917 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:07.977812 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:07.977817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:07.978110 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:07.978818 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:07.978839 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:07.980926 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:07.984546 (+ 3620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:07.984556 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:07.984577 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:07.985234 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:07.985568 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:07.985576 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:07.985582 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:07.985836 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:07.986307 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:07.986317 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:07.986746 (+ 429us) client_negotiation.cc:769] Sending connection context
0929 21:00:07.987039 (+ 293us) client_negotiation.cc:240] Negotiation successful
0929 21:00:07.987587 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":268,"threads_started":1}
I20260929 21:00:07.990573 6734 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38557
I20260929 21:00:07.990875 6734 heartbeater.cc:464] Registering TS with master...
I20260929 21:00:07.991556 6734 heartbeater.cc:511] Master 127.0.97.62:38557 requested a full tablet report, sending...
I20260929 21:00:07.993690 6545 ts_manager.cc:194] Registered new tserver with Master: 527d3ead14f6443b9fef7865f31d7227 (127.0.97.1:41637)
I20260929 21:00:07.995100 6545 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:60907
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260929 21:00:08.024039 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38557
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:00:08.337500 6740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:08.337778 6740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:08.337872 6740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:08.345073 6740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:08.345186 6740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:08.345299 6740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 21:00:08.354803 6740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:00:08.356027 6740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:08.357262 6740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:08.370121 6745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:08.370162 6748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:08.370085 6746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:08.374006 6740 server_base.cc:1034] running on GCE node
I20260929 21:00:08.374717 6740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:08.375803 6740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:08.377028 6740 hybrid_clock.cc:648] HybridClock initialized: now 1790715608376977 us; error 44 us; skew 500 ppm
I20260929 21:00:08.377385 6740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:00:08 dist-test-slave-1dms krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715608, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:00:08.381250 6740 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260929 21:00:08.382822 6740 webserver.cc:460] Webserver started at http://127.0.97.2:37581/ using document root <none> and password file <none>
I20260929 21:00:08.383504 6740 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:08.383651 6740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:08.383953 6740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:00:08.386912 6740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-1/data/instance:
uuid: "c5523b2f4ad14d37ba187ff45359077e"
format_stamp: "Formatted at 2026-09-29 21:00:08 on dist-test-slave-1dms"
server_key: "268434c6bfd56f6ca2b7f4fa667375e7"
server_key_iv: "8a2cf75c9bd5de13d380f97e5b542b03"
server_key_version: "encryptionkey@0"
I20260929 21:00:08.387668 6740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c5523b2f4ad14d37ba187ff45359077e"
format_stamp: "Formatted at 2026-09-29 21:00:08 on dist-test-slave-1dms"
server_key: "268434c6bfd56f6ca2b7f4fa667375e7"
server_key_iv: "8a2cf75c9bd5de13d380f97e5b542b03"
server_key_version: "encryptionkey@0"
I20260929 21:00:08.393186 6740 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 21:00:08.396657 6755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:08.398053 6740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:00:08.398249 6740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-1/wal
uuid: "c5523b2f4ad14d37ba187ff45359077e"
format_stamp: "Formatted at 2026-09-29 21:00:08 on dist-test-slave-1dms"
server_key: "268434c6bfd56f6ca2b7f4fa667375e7"
server_key_iv: "8a2cf75c9bd5de13d380f97e5b542b03"
server_key_version: "encryptionkey@0"
I20260929 21:00:08.398465 6740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:08.421300 6740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:08.426923 6740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:00:08.427209 6740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:08.428229 6740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:00:08.429541 6740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:00:08.429616 6740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:08.429700 6740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:00:08.429744 6740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:08.461359 6740 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40605
I20260929 21:00:08.461436 6867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40605 every 8 connection(s)
I20260929 21:00:08.462671 6740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
I20260929 21:00:08.463166 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 6740
I20260929 21:00:08.463364 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-1/wal/instance
I20260929 21:00:08.463991 388 external_mini_cluster.cc:1428] Setting key 0cae1eec95ff4546889dded04c595fcd
Sep 29 21:00:08 dist-test-slave-1dms krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715608, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 21:00:08.485245 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:08.465387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57379 (local address 127.0.97.62:38557)
0929 21:00:08.465552 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:08.465557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:08.467066 (+ 1509us) server_negotiation.cc:407] Connection header received
0929 21:00:08.468877 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:08.468882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:08.468970 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:08.469090 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:08.470575 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:08.471281 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:08.473814 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:08.474235 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:08.474343 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:08.478852 (+ 4509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:08.478857 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:08.478860 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:08.478894 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:08.481640 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:08.482707 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:08.482713 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:08.482716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:08.482873 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:08.483378 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:08.483381 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:08.483382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:08.483667 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:08.483775 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:08.484756 (+ 981us) server_negotiation.cc:299] Negotiation successful
0929 21:00:08.484995 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 21:00:08.486424 6870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:08.466092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38557 (local address 127.0.97.2:57379)
0929 21:00:08.466905 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:08.466946 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:08.468577 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:08.469326 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:08.469334 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:08.469848 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:08.470398 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:08.470414 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:08.471433 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:08.471445 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:08.473654 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:08.473673 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:08.474469 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:08.474477 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:08.474819 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:08.475570 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:08.475604 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:08.478586 (+ 2982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:08.481807 (+ 3221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:08.481820 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:08.481843 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:08.482533 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:08.483001 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:08.483008 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:08.483012 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:08.483264 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:08.483808 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:08.483819 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:08.484296 (+ 477us) client_negotiation.cc:769] Sending connection context
0929 21:00:08.484653 (+ 357us) client_negotiation.cc:240] Negotiation successful
0929 21:00:08.485257 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":325,"threads_started":1}
I20260929 21:00:08.487900 6868 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38557
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
I20260929 21:00:08.488294 6868 heartbeater.cc:464] Registering TS with master...
I20260929 21:00:08.489089 6868 heartbeater.cc:511] Master 127.0.97.62:38557 requested a full tablet report, sending...
I20260929 21:00:08.490658 6545 ts_manager.cc:194] Registered new tserver with Master: c5523b2f4ad14d37ba187ff45359077e (127.0.97.2:40605)
I20260929 21:00:08.491437 6545 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57379
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260929 21:00:08.514004 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38557
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:00:08.828119 6873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:08.828399 6873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:08.828570 6873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:08.835752 6873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:08.835855 6873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:08.836035 6873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 21:00:08.845495 6873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:00:08.846760 6873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:08.848037 6873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:08.860714 6881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:08.860668 6878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:08.860762 6879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:08.861542 6873 server_base.cc:1034] running on GCE node
I20260929 21:00:08.862102 6873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:08.862988 6873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:08.864207 6873 hybrid_clock.cc:648] HybridClock initialized: now 1790715608864162 us; error 43 us; skew 500 ppm
I20260929 21:00:08.864506 6873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:00:08 dist-test-slave-1dms krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715608, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:00:08.867470 6873 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260929 21:00:08.868903 6873 webserver.cc:460] Webserver started at http://127.0.97.3:39349/ using document root <none> and password file <none>
I20260929 21:00:08.869549 6873 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:08.869688 6873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:08.869973 6873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:00:08.872733 6873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-2/data/instance:
uuid: "34844c02e0f34d5fab45f43e4e5b9956"
format_stamp: "Formatted at 2026-09-29 21:00:08 on dist-test-slave-1dms"
server_key: "e671cc35f9409d3345e8183fae290d0c"
server_key_iv: "97811185d5c81081f530692bc9a034c5"
server_key_version: "encryptionkey@0"
I20260929 21:00:08.873540 6873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-2/wal/instance:
uuid: "34844c02e0f34d5fab45f43e4e5b9956"
format_stamp: "Formatted at 2026-09-29 21:00:08 on dist-test-slave-1dms"
server_key: "e671cc35f9409d3345e8183fae290d0c"
server_key_iv: "97811185d5c81081f530692bc9a034c5"
server_key_version: "encryptionkey@0"
I20260929 21:00:08.878400 6873 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 21:00:08.881726 6888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:08.882987 6873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 21:00:08.883159 6873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-2/wal
uuid: "34844c02e0f34d5fab45f43e4e5b9956"
format_stamp: "Formatted at 2026-09-29 21:00:08 on dist-test-slave-1dms"
server_key: "e671cc35f9409d3345e8183fae290d0c"
server_key_iv: "97811185d5c81081f530692bc9a034c5"
server_key_version: "encryptionkey@0"
I20260929 21:00:08.883369 6873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:08.900496 6873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:08.904620 6873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:00:08.904953 6873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:08.905922 6873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:00:08.907235 6873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:00:08.907346 6873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:08.907495 6873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:00:08.907567 6873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:08.939726 6873 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36861
I20260929 21:00:08.939810 7000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36861 every 8 connection(s)
I20260929 21:00:08.940969 6873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
I20260929 21:00:08.942685 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 6873
I20260929 21:00:08.942871 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-2/wal/instance
I20260929 21:00:08.943341 388 external_mini_cluster.cc:1428] Setting key cc5be61fd36ab7196fc2321584032726
Sep 29 21:00:08 dist-test-slave-1dms krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715608, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 21:00:08.959913 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:08.943450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48973 (local address 127.0.97.62:38557)
0929 21:00:08.943649 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:08.943655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:08.945017 (+ 1362us) server_negotiation.cc:407] Connection header received
0929 21:00:08.947097 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:08.947101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:08.947162 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:08.947243 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:08.948602 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:08.949456 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:08.950901 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:08.951269 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:08.951360 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:08.954546 (+ 3186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:08.954551 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:08.954554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:08.954584 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:08.956974 (+ 2390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:08.957849 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:08.957855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:08.957857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:08.958007 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:08.958451 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:08.958456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:08.958459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:08.958761 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:08.958871 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:08.959508 (+ 637us) server_negotiation.cc:299] Negotiation successful
0929 21:00:08.959658 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 21:00:08.960903 7003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:08.943987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38557 (local address 127.0.97.3:48973)
0929 21:00:08.944821 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:08.944915 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:08.946867 (+ 1952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:08.947438 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:08.947445 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:08.947920 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:08.948461 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:08.948476 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:08.949584 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:08.949588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:08.950780 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:08.950791 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:08.951446 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:08.951450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:08.951629 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:08.952218 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:08.952242 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:08.954361 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:08.957097 (+ 2736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:08.957105 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:08.957121 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:08.957722 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:08.958114 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:08.958121 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:08.958125 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:08.958340 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:08.958879 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:08.958885 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:08.959203 (+ 318us) client_negotiation.cc:769] Sending connection context
0929 21:00:08.959427 (+ 224us) client_negotiation.cc:240] Negotiation successful
0929 21:00:08.959784 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":285,"threads_started":1}
I20260929 21:00:08.962168 7001 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38557
I20260929 21:00:08.962455 7001 heartbeater.cc:464] Registering TS with master...
I20260929 21:00:08.963132 7001 heartbeater.cc:511] Master 127.0.97.62:38557 requested a full tablet report, sending...
I20260929 21:00:08.964429 6545 ts_manager.cc:194] Registered new tserver with Master: 34844c02e0f34d5fab45f43e4e5b9956 (127.0.97.3:36861)
I20260929 21:00:08.965256 6545 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48973
I20260929 21:00:08.970777 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
I20260929 21:00:08.998063 6734 heartbeater.cc:503] Master 127.0.97.62:38557 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260929 21:00:09.016224 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:44943
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38557,127.0.97.61:44943
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:00:09.335898 7006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:09.336175 7006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:09.336274 7006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:09.343441 7006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:00:09.343547 7006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:09.343592 7006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:09.343649 7006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:00:09.343688 7006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:00:09.353178 7006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38557,127.0.97.61:44943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:44943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
W20260929 21:00:09.353477 7006 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:38557,127.0.97.61:44943'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 21:00:09.354439 7006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:09.355643 7006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:09.367010 7012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:09.366955 7011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:09.367029 7014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:09.367760 7006 server_base.cc:1034] running on GCE node
I20260929 21:00:09.368533 7006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:09.369988 7006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:09.371219 7006 hybrid_clock.cc:648] HybridClock initialized: now 1790715609371140 us; error 59 us; skew 500 ppm
I20260929 21:00:09.371539 7006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:00:09 dist-test-slave-1dms krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715609, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:00:09.374691 7006 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260929 21:00:09.377950 7006 webserver.cc:460] Webserver started at http://127.0.97.61:44117/ using document root <none> and password file <none>
I20260929 21:00:09.378655 7006 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:09.378763 7006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:09.379076 7006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:00:09.381946 7006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-1/data/instance:
uuid: "b72fa9367117481cb47b95f56114e583"
format_stamp: "Formatted at 2026-09-29 21:00:09 on dist-test-slave-1dms"
server_key: "67478bdd708b010c907b2e76855e95ee"
server_key_iv: "cc5773e2415176b43d99c31557085514"
server_key_version: "encryptionkey@0"
I20260929 21:00:09.382669 7006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-1/wal/instance:
uuid: "b72fa9367117481cb47b95f56114e583"
format_stamp: "Formatted at 2026-09-29 21:00:09 on dist-test-slave-1dms"
server_key: "67478bdd708b010c907b2e76855e95ee"
server_key_iv: "cc5773e2415176b43d99c31557085514"
server_key_version: "encryptionkey@0"
I20260929 21:00:09.387720 7006 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 21:00:09.390987 7021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:09.392230 7006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 21:00:09.392390 7006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-1/wal
uuid: "b72fa9367117481cb47b95f56114e583"
format_stamp: "Formatted at 2026-09-29 21:00:09 on dist-test-slave-1dms"
server_key: "67478bdd708b010c907b2e76855e95ee"
server_key_iv: "cc5773e2415176b43d99c31557085514"
server_key_version: "encryptionkey@0"
I20260929 21:00:09.392581 7006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:09.420298 7006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:09.424491 7006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:00:09.424779 7006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:09.444383 7006 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:44943
I20260929 21:00:09.444360 7072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:44943 every 8 connection(s)
I20260929 21:00:09.445811 7006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-1/data/info.pb
I20260929 21:00:09.449644 7073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:00:09.454658 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 7006
I20260929 21:00:09.456111 7073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 21:00:09 dist-test-slave-1dms krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715609, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 29 21:00:09 dist-test-slave-1dms krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260929 21:00:09.476979 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:09.459093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60268 (local address 127.0.97.62:38557)
0929 21:00:09.459287 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:09.459293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:09.460052 (+ 759us) server_negotiation.cc:407] Connection header received
0929 21:00:09.460466 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:09.460471 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:09.460546 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:09.460643 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:09.462208 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:09.463398 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:09.465709 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:09.466304 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:09.466451 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:09.470063 (+ 3612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:09.470068 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:09.470071 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:09.470103 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:09.473506 (+ 3403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:09.474393 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:09.474397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:09.474399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:09.474550 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:09.475030 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:09.475033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:09.475036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:09.475456 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:09.475593 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:09.476185 (+ 592us) server_negotiation.cc:299] Negotiation successful
0929 21:00:09.476414 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 21:00:09.477849 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:09.459117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38557 (local address 127.0.0.1:60268)
0929 21:00:09.459902 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:09.459941 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:09.460214 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:09.460859 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:09.460866 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:09.461683 (+ 817us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:09.462031 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:09.462048 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:09.463546 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:09.463553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:09.465539 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:09.465553 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:09.466524 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:09.466817 (+ 293us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:09.466938 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:09.467548 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:09.467570 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:09.469868 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:09.473633 (+ 3765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:09.473642 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:09.473650 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:09.474237 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:09.474720 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:09.474724 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:09.474726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:09.474914 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:09.475580 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:09.475586 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:09.475792 (+ 206us) client_negotiation.cc:769] Sending connection context
0929 21:00:09.476072 (+ 280us) client_negotiation.cc:240] Negotiation successful
0929 21:00:09.476595 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":1,"thread_start_us":407,"threads_started":1}
I20260929 21:00:09.477866 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:09.456957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36530 (local address 127.0.97.61:44943)
0929 21:00:09.457754 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:09.457769 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:09.457816 (+ 47us) server_negotiation.cc:407] Connection header received
0929 21:00:09.459896 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:09.459919 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:09.460303 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:09.460518 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:09.462335 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:09.463862 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:09.465230 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:09.465930 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:09.466097 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:09.470418 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:09.470428 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:09.470443 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:09.470480 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:09.473873 (+ 3393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:09.474621 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:09.474629 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:09.474632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:09.474820 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:09.475215 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:09.475219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:09.475222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:09.475817 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:09.476021 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:09.476435 (+ 414us) server_negotiation.cc:299] Negotiation successful
0929 21:00:09.476711 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":79,"server-negotiator.queue_time_us":574,"spinlock_wait_cycles":29440,"thread_start_us":371,"threads_started":1}
I20260929 21:00:09.479661 7073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 21:00:09 dist-test-slave-1dms krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715609, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260929 21:00:09.491200 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:09.480293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44943 (local address 127.0.0.1:36536)
0929 21:00:09.480470 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:09.480489 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:09.480616 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:09.480912 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:09.480916 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:09.481231 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:09.481355 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:09.481362 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:09.482277 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:09.482282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:09.483118 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:09.483127 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:09.483759 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:09.483764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:09.483979 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:09.484452 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:09.484467 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:09.486478 (+ 2011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:09.489343 (+ 2865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:09.489347 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:09.489349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:09.489671 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:09.489961 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:09.489964 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:09.489966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:09.490077 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:09.490520 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:09.490523 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:09.490626 (+ 103us) client_negotiation.cc:769] Sending connection context
0929 21:00:09.490729 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 21:00:09.490926 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":1}
I20260929 21:00:09.491281 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:09.480324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36536 (local address 127.0.97.61:44943)
0929 21:00:09.480470 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:09.480475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:09.480561 (+ 86us) server_negotiation.cc:407] Connection header received
0929 21:00:09.480715 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:09.480721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:09.480791 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:09.480901 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:09.481479 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:09.482152 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:09.483247 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:09.483624 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:09.483759 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:09.486678 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:09.486683 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:09.486686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:09.486714 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:09.489228 (+ 2514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:09.489767 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:09.489771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:09.489773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:09.489874 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:09.490173 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:09.490176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:09.490178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:09.490414 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:09.490519 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:09.490817 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 21:00:09.491031 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":101,"server-negotiator.queue_time_us":40}
I20260929 21:00:09.494179 6868 heartbeater.cc:503] Master 127.0.97.62:38557 was elected leader, sending a full tablet report...
I20260929 21:00:09.496212 7073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583: Bootstrap starting.
I20260929 21:00:09.499290 7073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583: Neither blocks nor log segments found. Creating new log.
I20260929 21:00:09.500320 7073 log.cc:824] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583: Log is configured to *not* fsync() on all Append() calls
I20260929 21:00:09.502940 7073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583: No bootstrap required, opened a new log
I20260929 21:00:09.509723 7073 raft_consensus.cc:348] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } }
I20260929 21:00:09.510094 7073 raft_consensus.cc:374] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:00:09.510217 7073 raft_consensus.cc:729] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b72fa9367117481cb47b95f56114e583, State: Initialized, Role: FOLLOWER
I20260929 21:00:09.510814 7073 consensus_queue.cc:260] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } }
I20260929 21:00:09.511584 7079 sys_catalog.cc:455] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } } }
I20260929 21:00:09.511904 7079 sys_catalog.cc:458] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 21:00:09.512492 7073 sys_catalog.cc:564] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 21:00:09.522908 7090 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 21:00:09.523044 7090 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 21:00:09.525470 7006 master_runner.cc:428] Detected that this master b72fa9367117481cb47b95f56114e583 is joining an existing cluster
I20260929 21:00:09.525565 7006 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:44943
I20260929 21:00:09.546569 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:09.529894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36546 (local address 127.0.97.61:44943)
0929 21:00:09.530032 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:09.530038 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:09.532361 (+ 2323us) server_negotiation.cc:407] Connection header received
0929 21:00:09.532555 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:09.532559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:09.532617 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:09.532674 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:09.533911 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:09.535052 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:09.536976 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:09.537479 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:09.537613 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:09.539917 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:09.539924 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:09.539929 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:09.539961 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:09.543737 (+ 3776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:09.544489 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:09.544497 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:09.544501 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:09.544676 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:09.545114 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:09.545119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:09.545122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:09.545439 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:09.545584 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:09.545937 (+ 353us) server_negotiation.cc:299] Negotiation successful
0929 21:00:09.546158 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":14,"server-negotiator.queue_time_us":57,"spinlock_wait_cycles":2816}
I20260929 21:00:09.547011 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:09.530244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60284 (local address 127.0.97.62:38557)
0929 21:00:09.530436 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:09.530442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:09.530461 (+ 19us) server_negotiation.cc:407] Connection header received
0929 21:00:09.530517 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:09.530523 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:09.530590 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:09.530695 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:09.532458 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:09.533636 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:09.535688 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:09.536300 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:09.536390 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:09.539917 (+ 3527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:09.539924 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:09.539929 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:09.539962 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:09.543761 (+ 3799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:09.544669 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:09.544674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:09.544678 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:09.544882 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:09.545289 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:09.545297 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:09.545303 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:09.545704 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:09.545841 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:09.546322 (+ 481us) server_negotiation.cc:299] Negotiation successful
0929 21:00:09.546563 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 21:00:09.547451 7097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:09.529473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44943 (local address 127.0.0.1:36546)
0929 21:00:09.532278 (+ 2805us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:09.532296 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:09.532441 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:09.532865 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:09.532869 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:09.533465 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:09.533757 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:09.533778 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:09.535244 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:09.535249 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:09.536784 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:09.536795 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:09.537662 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:09.537667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:09.537847 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:09.538616 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:09.538642 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:09.539728 (+ 1086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:09.543890 (+ 4162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:09.543896 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:09.543904 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:09.544377 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:09.544789 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:09.544794 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:09.544798 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:09.545001 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:09.545551 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:09.545555 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:09.545697 (+ 142us) client_negotiation.cc:769] Sending connection context
0929 21:00:09.546766 (+ 1069us) client_negotiation.cc:240] Negotiation successful
0929 21:00:09.547006 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2663,"spinlock_wait_cycles":44160,"thread_start_us":1815,"threads_started":1}
I20260929 21:00:09.547533 7096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:09.529326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38557 (local address 127.0.0.1:60284)
0929 21:00:09.530037 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:09.530055 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:09.530182 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:09.530722 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:09.530727 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:09.531825 (+ 1098us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:09.532295 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:09.532310 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:09.533810 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:09.533818 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:09.535527 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:09.535546 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:09.536676 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:09.536683 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:09.536964 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:09.538433 (+ 1469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:09.538462 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:09.539727 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:09.543888 (+ 4161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:09.543894 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:09.543898 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:09.544539 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:09.544990 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:09.544996 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:09.545000 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:09.545168 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:09.545850 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:09.545857 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:09.546035 (+ 178us) client_negotiation.cc:769] Sending connection context
0929 21:00:09.546195 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 21:00:09.546459 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":2,"spinlock_wait_cycles":5376,"thread_start_us":1501,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Sep 29 21:00:09 dist-test-slave-1dms krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715607, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260929 21:00:09.568882 7099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:09.553114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44943 (local address 127.0.0.1:36556)
0929 21:00:09.553782 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:09.553834 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:09.554007 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:09.554623 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:09.554630 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:09.555497 (+ 867us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:09.555765 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:09.555779 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:09.557197 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:09.557203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:09.558383 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:09.558394 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:09.559406 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:09.559410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:09.559544 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:09.560197 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:09.560234 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:09.562599 (+ 2365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:09.565929 (+ 3330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:09.565940 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:09.565948 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:09.566467 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:09.566856 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:09.566864 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:09.566869 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:09.566998 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:09.567638 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:09.567649 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:09.567872 (+ 223us) client_negotiation.cc:769] Sending connection context
0929 21:00:09.568081 (+ 209us) client_negotiation.cc:240] Negotiation successful
0929 21:00:09.568402 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":382,"threads_started":1}
I20260929 21:00:09.568928 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:09.553456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36556 (local address 127.0.97.61:44943)
0929 21:00:09.554240 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:09.554246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:09.554265 (+ 19us) server_negotiation.cc:407] Connection header received
0929 21:00:09.554363 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:09.554370 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:09.554443 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:09.554540 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:09.555927 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:09.557039 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:09.558526 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:09.559234 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:09.559367 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:09.562782 (+ 3415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:09.562787 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:09.562791 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:09.562819 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:09.565789 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:09.566615 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:09.566619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:09.566621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:09.566745 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:09.567104 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:09.567111 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:09.567115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:09.567487 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:09.567668 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:09.568210 (+ 542us) server_negotiation.cc:299] Negotiation successful
0929 21:00:09.568468 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651}
I20260929 21:00:09.571357 6545 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:44943: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b72fa9367117481cb47b95f56114e583"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 44943
}
attrs {
promote: true
}
}
dest_uuid: "4b9bcd97294f4ef68830d77516fb1754"
cas_config_opid_index: -1
I20260929 21:00:09.572487 6545 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: true } }
I20260929 21:00:09.583313 7048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:00:09.582782 7102 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b72fa9367117481cb47b95f56114e583 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: true } } }
I20260929 21:00:09.587447 7102 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:44943
I20260929 21:00:09.587622 7048 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [term 1 FOLLOWER]: Refusing update from remote peer 4b9bcd97294f4ef68830d77516fb1754: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 21:00:09.588089 7102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4b9bcd97294f4ef68830d77516fb1754" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: true } } }
I20260929 21:00:09.588449 7102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [sys.catalog]: This master's current role is: LEADER
I20260929 21:00:09.588618 7006 master.cc:537] Master@127.0.97.61:44943 shutting down...
I20260929 21:00:09.589273 7102 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/oryx_127.0.97.60.keytab.
W20260929 21:00:09.592643 6530 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260929 21:00:09.593144 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44957
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38557,127.0.97.61:44943,127.0.97.60:44957
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:00:09.594532 6530 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 -> Peer b72fa9367117481cb47b95f56114e583 (127.0.97.61:44943): Couldn't send request to peer b72fa9367117481cb47b95f56114e583. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 21:00:09.602864 7006 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:00:09.603330 7006 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:00:09.603533 7006 tablet_replica.cc:330] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583: stopping tablet replica
I20260929 21:00:09.609588 7006 master.cc:552] Master@127.0.97.61:44943 shutdown complete.
I20260929 21:00:09.609695 7006 master_runner.cc:305] Clearing existing system tablet
I20260929 21:00:09.610062 7006 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:09.610168 7006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:09.612403 7006 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 21:00:09.620069 7108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:09.620359 7006 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:09.620658 7006 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.007s sys 0.000s
I20260929 21:00:09.620748 7006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-1/wal
uuid: "b72fa9367117481cb47b95f56114e583"
format_stamp: "Formatted at 2026-09-29 21:00:09 on dist-test-slave-1dms"
server_key: "67478bdd708b010c907b2e76855e95ee"
server_key_iv: "cc5773e2415176b43d99c31557085514"
server_key_version: "encryptionkey@0"
I20260929 21:00:09.624104 7006 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:00:09.629137 7006 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 21:00:09.629315 7006 log.cc:1196] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 21:00:09.629731 7006 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583: Deleting consensus metadata
I20260929 21:00:09.629976 7006 master_runner.cc:315] Copying system tablet from 127.0.97.62:38557
I20260929 21:00:09.634117 7006 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:38557
I20260929 21:00:09.648276 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:09.634920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60298 (local address 127.0.97.62:38557)
0929 21:00:09.635148 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:09.635158 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:09.637393 (+ 2235us) server_negotiation.cc:407] Connection header received
0929 21:00:09.637607 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:09.637612 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:09.637701 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:09.637809 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:09.638761 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:09.639477 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:09.640669 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:09.641190 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:09.641320 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:09.643132 (+ 1812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:09.643138 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:09.643143 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:09.643183 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:09.645804 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:09.646418 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:09.646423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:09.646427 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:09.646547 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:09.646898 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:09.646906 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:09.646909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:09.647172 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:09.647328 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:09.647692 (+ 364us) server_negotiation.cc:299] Negotiation successful
0929 21:00:09.647917 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 21:00:09.648394 7114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:09.634835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38557 (local address 127.0.0.1:60298)
0929 21:00:09.637269 (+ 2434us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:09.637301 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:09.637464 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:09.637833 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:09.637838 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:09.638255 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:09.638470 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:09.638479 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:09.639616 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:09.639622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:09.640528 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:09.640539 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:09.641350 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:09.641355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:09.641569 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:09.642240 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:09.642257 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:09.642933 (+ 676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:09.645940 (+ 3007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:09.645947 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:09.645951 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:09.646297 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:09.646651 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:09.646657 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:09.646661 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:09.646796 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:09.647295 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:09.647300 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:09.647470 (+ 170us) client_negotiation.cc:769] Sending connection context
0929 21:00:09.647609 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 21:00:09.647929 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2314,"thread_start_us":2231,"threads_started":1}
I20260929 21:00:09.649463 6565 tablet_copy_service.cc:140] P 4b9bcd97294f4ef68830d77516fb1754: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b72fa9367117481cb47b95f56114e583 ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:60298)
I20260929 21:00:09.649720 6565 tablet_copy_service.cc:161] P 4b9bcd97294f4ef68830d77516fb1754: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b72fa9367117481cb47b95f56114e583 at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:60298: session id = b72fa9367117481cb47b95f56114e583-00000000000000000000000000000000
I20260929 21:00:09.652743 6565 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 21:00:09.654644 7006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 21:00:09.659382 7006 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583: tablet copy: Starting download of 0 data blocks...
I20260929 21:00:09.659685 7006 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583: tablet copy: Starting download of 1 WAL segments...
I20260929 21:00:09.662098 7006 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 21:00:09.665267 6565 tablet_copy_service.cc:342] P 4b9bcd97294f4ef68830d77516fb1754: Request end of tablet copy session b72fa9367117481cb47b95f56114e583-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:60298
I20260929 21:00:09.665464 6565 tablet_copy_service.cc:433] P 4b9bcd97294f4ef68830d77516fb1754: ending tablet copy session b72fa9367117481cb47b95f56114e583-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b72fa9367117481cb47b95f56114e583
I20260929 21:00:09.674921 7006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:09.677512 7115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:09.687963 7116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:09.695977 7118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:09.696475 7006 server_base.cc:1034] running on GCE node
I20260929 21:00:09.696871 7006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:09.697530 7006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:09.698741 7006 hybrid_clock.cc:648] HybridClock initialized: now 1790715609698669 us; error 52 us; skew 500 ppm
I20260929 21:00:09.698979 7006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:00:09 dist-test-slave-1dms krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715609, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:00:09.700994 7006 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260929 21:00:09.702675 7006 webserver.cc:460] Webserver started at http://127.0.97.61:37215/ using document root <none> and password file <none>
I20260929 21:00:09.703015 7006 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:09.703111 7006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:09.705709 7006 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 21:00:09.707335 7125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:09.707775 7006 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260929 21:00:09.707932 7006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-1/wal
uuid: "b72fa9367117481cb47b95f56114e583"
format_stamp: "Formatted at 2026-09-29 21:00:09 on dist-test-slave-1dms"
server_key: "67478bdd708b010c907b2e76855e95ee"
server_key_iv: "cc5773e2415176b43d99c31557085514"
server_key_version: "encryptionkey@0"
I20260929 21:00:09.708084 7006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:09.722815 7006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:09.726437 7006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:09.754477 7006 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:44943
I20260929 21:00:09.754500 7176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:44943 every 8 connection(s)
I20260929 21:00:09.755362 7006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-1/data/info.pb
I20260929 21:00:09.762113 7177 sys_catalog.cc:263] Verifying existing consensus state
I20260929 21:00:09.764810 7177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583: Bootstrap starting.
I20260929 21:00:09.776612 7177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 21:00:09.777069 7177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583: Bootstrap complete.
I20260929 21:00:09.777878 7177 raft_consensus.cc:348] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: true } }
I20260929 21:00:09.778102 7177 raft_consensus.cc:729] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b72fa9367117481cb47b95f56114e583, State: Initialized, Role: LEARNER
I20260929 21:00:09.778334 7177 consensus_queue.cc:260] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: true } }
I20260929 21:00:09.779095 7180 sys_catalog.cc:455] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: true } } }
I20260929 21:00:09.779407 7177 sys_catalog.cc:564] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:00:09.779445 7180 sys_catalog.cc:458] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [sys.catalog]: This master's current role is: LEARNER
I20260929 21:00:09.790313 7191 catalog_manager.cc:1259] Loaded cluster ID: 1d78b66f93774de588144898a42be9a5
I20260929 21:00:09.790423 7191 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583: loading cluster ID for follower catalog manager: success
I20260929 21:00:09.792624 7191 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583: acquiring CA information for follower catalog manager: success
I20260929 21:00:09.794500 7191 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 21:00:09.968165 7001 heartbeater.cc:503] Master 127.0.97.62:38557 was elected leader, sending a full tablet report...
W20260929 21:00:09.991660 7106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:09.991930 7106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:09.992086 7106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:09.999639 7106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:00:09.999739 7106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:09.999820 7106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:09.999922 7106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:00:10.000000 7106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:00:10.010792 7106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38557,127.0.97.61:44943,127.0.97.60:44957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:00:10.012122 7106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:10.013518 7106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:10.025626 7199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:10.025806 7196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:10.025895 7197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:10.026572 7106 server_base.cc:1034] running on GCE node
I20260929 21:00:10.027395 7106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:10.028774 7106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:10.029999 7106 hybrid_clock.cc:648] HybridClock initialized: now 1790715610029958 us; error 40 us; skew 500 ppm
I20260929 21:00:10.030301 7106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:00:10 dist-test-slave-1dms krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715610, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:00:10.033478 7106 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260929 21:00:10.034883 7106 webserver.cc:460] Webserver started at http://127.0.97.60:33815/ using document root <none> and password file <none>
I20260929 21:00:10.035881 7106 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:10.035983 7106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:10.036232 7106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:00:10.039143 7106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-2/data/instance:
uuid: "46eee36a2fe34cd8b2582f89c8f11aff"
format_stamp: "Formatted at 2026-09-29 21:00:10 on dist-test-slave-1dms"
server_key: "c2ac0b7704585d7de11e7132569708e2"
server_key_iv: "d499f6a324d244f0d90045b477936441"
server_key_version: "encryptionkey@0"
I20260929 21:00:10.039831 7106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-2/wal/instance:
uuid: "46eee36a2fe34cd8b2582f89c8f11aff"
format_stamp: "Formatted at 2026-09-29 21:00:10 on dist-test-slave-1dms"
server_key: "c2ac0b7704585d7de11e7132569708e2"
server_key_iv: "d499f6a324d244f0d90045b477936441"
server_key_version: "encryptionkey@0"
I20260929 21:00:10.044760 7106 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 21:00:10.048017 7206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:10.049422 7106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 21:00:10.049600 7106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-2/wal
uuid: "46eee36a2fe34cd8b2582f89c8f11aff"
format_stamp: "Formatted at 2026-09-29 21:00:10 on dist-test-slave-1dms"
server_key: "c2ac0b7704585d7de11e7132569708e2"
server_key_iv: "d499f6a324d244f0d90045b477936441"
server_key_version: "encryptionkey@0"
I20260929 21:00:10.049800 7106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:10.075742 7106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:10.080621 7106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:00:10.080925 7106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:10.099625 7106 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:44957
I20260929 21:00:10.099653 7257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:44957 every 8 connection(s)
I20260929 21:00:10.101058 7106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-2/data/info.pb
I20260929 21:00:10.104976 7258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:00:10.109421 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 7106
I20260929 21:00:10.112864 7258 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 21:00:10 dist-test-slave-1dms krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 29 21:00:10 dist-test-slave-1dms krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715610, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 21:00:10.130939 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:10.116645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60306 (local address 127.0.97.62:38557)
0929 21:00:10.116828 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:10.116852 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:10.117407 (+ 555us) server_negotiation.cc:407] Connection header received
0929 21:00:10.117599 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:10.117605 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:10.117746 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:10.117862 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:10.118911 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.119597 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.121521 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.121902 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.122024 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:10.125456 (+ 3432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:10.125460 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:10.125463 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:10.125491 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:10.128101 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:10.128743 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:10.128747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:10.128749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:10.128932 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:10.129329 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:10.129333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:10.129336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:10.129630 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:10.129766 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:10.130350 (+ 584us) server_negotiation.cc:299] Negotiation successful
0929 21:00:10.130566 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 21:00:10.132239 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:10.111500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60406 (local address 127.0.97.60:44957)
0929 21:00:10.112236 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:10.112247 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:10.112285 (+ 38us) server_negotiation.cc:407] Connection header received
0929 21:00:10.113823 (+ 1538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:10.113844 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:10.114213 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:10.114522 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:10.116273 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.118288 (+ 2015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.119491 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.120522 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.120799 (+ 277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:10.125120 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:10.125133 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:10.125147 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:10.125195 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:10.128002 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:10.128626 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:10.128634 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:10.128637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:10.128811 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:10.129254 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:10.129284 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:10.129288 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:10.129927 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:10.130251 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:10.130607 (+ 356us) server_negotiation.cc:299] Negotiation successful
0929 21:00:10.131127 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"spinlock_wait_cycles":12672,"thread_start_us":354,"threads_started":1}
I20260929 21:00:10.132287 7262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:10.116539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38557 (local address 127.0.0.1:60306)
0929 21:00:10.117287 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:10.117318 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:10.117480 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:10.117954 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:10.117961 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:10.118544 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:10.118764 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:10.118781 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.119742 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.119751 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:10.121382 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:10.121392 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.122116 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.122122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:10.122267 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:10.122921 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:10.122951 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:10.125296 (+ 2345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:10.128218 (+ 2922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:10.128225 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:10.128233 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:10.128626 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:10.129011 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:10.129017 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:10.129021 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:10.129203 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:10.129743 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:10.129749 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:10.129912 (+ 163us) client_negotiation.cc:769] Sending connection context
0929 21:00:10.131361 (+ 1449us) client_negotiation.cc:240] Negotiation successful
0929 21:00:10.131679 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":381,"threads_started":1}
I20260929 21:00:10.135226 7258 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } has no permanent_uuid. Determining permanent_uuid...
I20260929 21:00:10.140465 7265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:10.133358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36564 (local address 127.0.97.61:44943)
0929 21:00:10.134062 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:10.134070 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:10.134090 (+ 20us) server_negotiation.cc:407] Connection header received
0929 21:00:10.134161 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:10.134167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:10.134258 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:10.134391 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:00:10.134809 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.136268 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.138449 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.139639 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.139790 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:10.139990 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:10.140041 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 21:00:10.140167 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":443,"threads_started":1}
I20260929 21:00:10.140460 7264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:10.132821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44943 (local address 127.0.0.1:36564)
0929 21:00:10.133755 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:10.133776 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:10.133896 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:10.134414 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:10.134419 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:10.134446 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:00:10.134661 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:10.134672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.136410 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.136416 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:10.138290 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:10.138303 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.139806 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.139810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:10.139923 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:10.139960 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 21:00:10.140020 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 21:00:10.140107 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"thread_start_us":221,"threads_started":1}
I20260929 21:00:10.141759 7152 raft_consensus.cc:1203] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260929 21:00:10.142545 7180 sys_catalog.cc:455] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b9bcd97294f4ef68830d77516fb1754. Latest consensus state: current_term: 1 leader_uuid: "4b9bcd97294f4ef68830d77516fb1754" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: true } } }
I20260929 21:00:10.142786 7180 sys_catalog.cc:458] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [sys.catalog]: This master's current role is: LEARNER
Sep 29 21:00:10 dist-test-slave-1dms krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715610, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260929 21:00:10.149629 7262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:10.135939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44943 (local address 127.0.0.1:36578)
0929 21:00:10.136097 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:10.136115 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:10.136218 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:10.136914 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:10.136917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:10.137221 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:10.137340 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:10.137347 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.138712 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.138717 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:10.140078 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:10.140091 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.141097 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.141105 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:10.141310 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:10.141872 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:10.141897 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:10.144732 (+ 2835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:10.147699 (+ 2967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:10.147703 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:10.147706 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:10.148051 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:10.148363 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:10.148366 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:10.148368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:10.148484 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:10.148973 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:10.148976 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:10.149090 (+ 114us) client_negotiation.cc:769] Sending connection context
0929 21:00:10.149216 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 21:00:10.149417 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 21:00:10.149693 7266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:10.136027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36578 (local address 127.0.97.61:44943)
0929 21:00:10.136579 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:10.136586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:10.136606 (+ 20us) server_negotiation.cc:407] Connection header received
0929 21:00:10.136651 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:10.136656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:10.136725 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:10.136871 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:10.137470 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.138585 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.140215 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.140966 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.141054 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:10.144958 (+ 3904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:10.144963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:10.144966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:10.144995 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:10.147575 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:10.148145 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:10.148149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:10.148151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:10.148270 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:10.148567 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:10.148570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:10.148572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:10.148805 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:10.148994 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:10.149273 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 21:00:10.149420 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":304,"threads_started":1}
I20260929 21:00:10.150609 7258 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44957 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 21:00:10 dist-test-slave-1dms krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715610, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260929 21:00:10.161907 7262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:10.151255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44957 (local address 127.0.0.1:60410)
0929 21:00:10.151402 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:10.151417 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:10.151521 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:10.151770 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:10.151774 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:10.152112 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:10.152230 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:10.152237 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.153207 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.153213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:10.154079 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:10.154089 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.154715 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.154720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:10.154908 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:10.155378 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:10.155393 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:10.157220 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:10.159673 (+ 2453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:10.159680 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:10.159684 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:10.159997 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:10.160333 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:10.160338 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:10.160342 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:10.160476 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:10.160974 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:10.160979 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:10.161106 (+ 127us) client_negotiation.cc:769] Sending connection context
0929 21:00:10.161232 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 21:00:10.161494 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 21:00:10.161937 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:10.151407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60410 (local address 127.0.97.60:44957)
0929 21:00:10.151535 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:10.151540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:10.151558 (+ 18us) server_negotiation.cc:407] Connection header received
0929 21:00:10.151604 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:10.151609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:10.151663 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:10.151756 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:10.152360 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.153098 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.154203 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.154585 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.154711 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:10.157393 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:10.157398 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:10.157401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:10.157425 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:10.159555 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:10.160102 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:10.160110 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:10.160113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:10.160230 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:10.160572 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:10.160580 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:10.160583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:10.160818 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:10.161007 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:10.161316 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 21:00:10.161611 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":121,"server-negotiator.queue_time_us":44}
I20260929 21:00:10.166452 7258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff: Bootstrap starting.
I20260929 21:00:10.169862 7258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff: Neither blocks nor log segments found. Creating new log.
I20260929 21:00:10.170825 7258 log.cc:824] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff: Log is configured to *not* fsync() on all Append() calls
I20260929 21:00:10.173250 7258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff: No bootstrap required, opened a new log
I20260929 21:00:10.180162 7258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } } peers { permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44957 } }
I20260929 21:00:10.180523 7258 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:00:10.180640 7258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46eee36a2fe34cd8b2582f89c8f11aff, State: Initialized, Role: FOLLOWER
I20260929 21:00:10.181295 7258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } } peers { permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44957 } }
I20260929 21:00:10.182449 7268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } } peers { permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44957 } } }
I20260929 21:00:10.182713 7268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [sys.catalog]: This master's current role is: FOLLOWER
I20260929 21:00:10.183440 7258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [sys.catalog]: configured and running, proceeding with master startup.
W20260929 21:00:10.192438 7279 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 21:00:10.192576 7279 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 21:00:10.196979 7106 master_runner.cc:428] Detected that this master 46eee36a2fe34cd8b2582f89c8f11aff is joining an existing cluster
I20260929 21:00:10.197068 7106 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:44957
I20260929 21:00:10.223827 7266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:10.200722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36588 (local address 127.0.97.61:44943)
0929 21:00:10.200910 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:10.200917 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:10.206392 (+ 5475us) server_negotiation.cc:407] Connection header received
0929 21:00:10.206570 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:10.206576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:10.206722 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:10.206854 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:10.208638 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.209717 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.211659 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.212338 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.212465 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:10.215012 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:10.215020 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:10.215024 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:10.215059 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:10.220565 (+ 5506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:10.221549 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:10.221554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:10.221558 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:10.221703 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:10.222158 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:10.222166 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:10.222172 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:10.222521 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:10.222651 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:10.223255 (+ 604us) server_negotiation.cc:299] Negotiation successful
0929 21:00:10.223482 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 21:00:10.224879 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:10.201247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60314 (local address 127.0.97.62:38557)
0929 21:00:10.201456 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:10.201463 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:10.202111 (+ 648us) server_negotiation.cc:407] Connection header received
0929 21:00:10.202446 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:10.202455 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:10.202559 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:10.202645 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:10.207191 (+ 4546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.208556 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.210620 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.211248 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.211345 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:10.215040 (+ 3695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:10.215046 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:10.215050 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:10.215221 (+ 171us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:10.220223 (+ 5002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:10.221966 (+ 1743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:10.221973 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:10.221977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:10.222114 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:10.222873 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:10.222880 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:10.222886 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:10.223237 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:10.223365 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:10.224272 (+ 907us) server_negotiation.cc:299] Negotiation successful
0929 21:00:10.224484 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 21:00:10.224910 7287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:10.200895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44943 (local address 127.0.0.1:36588)
0929 21:00:10.206290 (+ 5395us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:10.206313 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:10.206440 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:10.207106 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:10.207110 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:10.208127 (+ 1017us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:10.208418 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:10.208429 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.209953 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.209958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:10.211510 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:10.211522 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.212695 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.212703 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:10.212948 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:10.213657 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:10.213689 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:10.214836 (+ 1147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:10.220956 (+ 6120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:10.220961 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:10.220968 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:10.221406 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:10.221847 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:10.221851 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:10.221853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:10.222038 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:10.222732 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:10.222736 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:10.222964 (+ 228us) client_negotiation.cc:769] Sending connection context
0929 21:00:10.224068 (+ 1104us) client_negotiation.cc:240] Negotiation successful
0929 21:00:10.224385 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5235,"thread_start_us":16373,"threads_started":1}
I20260929 21:00:10.226408 7285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:10.200683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38557 (local address 127.0.0.1:60314)
0929 21:00:10.202016 (+ 1333us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:10.202041 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:10.202208 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:10.202901 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:10.202909 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:10.206649 (+ 3740us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:10.207030 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:10.207044 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.208725 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.208730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:10.210469 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:10.210481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.211606 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.211614 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:10.211783 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:10.212529 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:10.212594 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:10.214724 (+ 2130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:10.221335 (+ 6611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:10.221341 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:10.221345 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:10.221839 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:10.222249 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:10.222254 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:10.222257 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:10.222736 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:10.223442 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:10.223447 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:10.223650 (+ 203us) client_negotiation.cc:769] Sending connection context
0929 21:00:10.225553 (+ 1903us) client_negotiation.cc:240] Negotiation successful
0929 21:00:10.225893 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1192,"thread_start_us":1009,"threads_started":1}
I20260929 21:00:10.233419 7286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:10.202138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44957 (local address 127.0.0.1:60414)
0929 21:00:10.218281 (+ 16143us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:10.218305 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:10.218458 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:10.218855 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:10.218862 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:10.219508 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:10.219819 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:10.219830 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.221313 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.221318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:10.223270 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:10.223284 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.224510 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.224520 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:10.224676 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:10.225376 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:10.225403 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:10.226535 (+ 1132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:10.230162 (+ 3627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:10.230168 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:10.230172 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:10.230673 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:10.231191 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:10.231198 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:10.231201 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:10.231319 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:10.232363 (+ 1044us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:10.232370 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:10.232483 (+ 113us) client_negotiation.cc:769] Sending connection context
0929 21:00:10.232620 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 21:00:10.232930 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15963,"thread_start_us":3848,"threads_started":1}
I20260929 21:00:10.233395 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:10.202413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60414 (local address 127.0.97.60:44957)
0929 21:00:10.202582 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:10.202588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:10.218375 (+ 15787us) server_negotiation.cc:407] Connection header received
0929 21:00:10.218571 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:10.218576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:10.218635 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:10.218695 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:10.219963 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.221091 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.223410 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.224108 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.224201 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:10.226730 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:10.226737 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:10.226744 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:10.226781 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:10.230018 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:10.230957 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:10.230965 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:10.230968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:10.231098 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:10.231422 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:10.231429 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:10.231433 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:10.231812 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:10.232327 (+ 515us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:10.232701 (+ 374us) server_negotiation.cc:299] Negotiation successful
0929 21:00:10.233056 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":166,"server-negotiator.queue_time_us":52}
I20260929 21:00:10.238835 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 6606
Sep 29 21:00:10 dist-test-slave-1dms krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715607, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260929 21:00:10.247181 7264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:10.231818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44957 (local address 127.0.0.1:60428)
0929 21:00:10.232059 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:10.232079 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:10.232169 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:10.234605 (+ 2436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:10.234610 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:10.235043 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:10.235228 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:10.235237 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.236306 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.236312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:10.237443 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:10.237458 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.238232 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.238238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:10.238410 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:10.239026 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:10.239050 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:10.241549 (+ 2499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:10.244570 (+ 3021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:10.244575 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:10.244579 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:10.245108 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:10.245502 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:10.245507 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:10.245511 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:10.245689 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:10.246295 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:10.246301 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:10.246474 (+ 173us) client_negotiation.cc:769] Sending connection context
0929 21:00:10.246621 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 21:00:10.246887 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 21:00:10.248072 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:10.232005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60428 (local address 127.0.97.60:44957)
0929 21:00:10.234220 (+ 2215us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:10.234225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:10.234241 (+ 16us) server_negotiation.cc:407] Connection header received
0929 21:00:10.234327 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:10.234336 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:10.234448 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:10.235344 (+ 896us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:10.235501 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.236165 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.237603 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.238070 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.238222 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:10.241711 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:10.241717 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:10.241721 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:10.241758 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:10.244434 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:10.245226 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:10.245232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:10.245235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:10.245372 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:10.245824 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:10.245830 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:10.245834 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:10.246159 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:10.246294 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:10.247662 (+ 1368us) server_negotiation.cc:299] Negotiation successful
0929 21:00:10.247833 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2119,"thread_start_us":5169,"threads_started":1}
Sep 29 21:00:10 dist-test-slave-1dms krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260929 21:00:10.252146 6545 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:44957: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 44957
}
attrs {
promote: true
}
}
dest_uuid: "4b9bcd97294f4ef68830d77516fb1754"
cas_config_opid_index: 5
I20260929 21:00:10.253273 6545 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: true } } peers { permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44957 } attrs { promote: true } }
I20260929 21:00:10.255108 7263 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 46eee36a2fe34cd8b2582f89c8f11aff (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: true } } peers { permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44957 } attrs { promote: true } } }
I20260929 21:00:10.255692 7152 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [term 1 LEARNER]: Refusing update from remote peer 4b9bcd97294f4ef68830d77516fb1754: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 21:00:10.257441 7233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:00:10.260133 7289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:10.239780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46852 (local address 127.0.97.1:41637)
0929 21:00:10.240366 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:10.240376 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:10.240407 (+ 31us) server_negotiation.cc:407] Connection header received
0929 21:00:10.240474 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:10.240483 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:10.240767 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:10.241060 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:10.242478 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.244033 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.245712 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.246543 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.246690 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:10.252214 (+ 5524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:10.252225 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:10.252240 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:10.252281 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:10.256099 (+ 3818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:10.256952 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:10.256962 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:10.256965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:10.257113 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:10.257549 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:10.257555 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:10.257560 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:10.258082 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:10.258359 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:10.259452 (+ 1093us) server_negotiation.cc:299] Negotiation successful
0929 21:00:10.259713 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":287,"threads_started":1}
I20260929 21:00:10.261121 7233 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [term 1 FOLLOWER]: Refusing update from remote peer 4b9bcd97294f4ef68830d77516fb1754: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 21:00:10.262240 6729 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:00:10.277256 7291 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 21:00:10.289944 7263 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:44957
I20260929 21:00:10.290712 7106 master.cc:537] Master@127.0.97.60:44957 shutting down...
I20260929 21:00:10.291957 7152 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 46eee36a2fe34cd8b2582f89c8f11aff (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: true } } peers { permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44957 } attrs { promote: true } } }
I20260929 21:00:10.296213 7180 sys_catalog.cc:455] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b9bcd97294f4ef68830d77516fb1754" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: true } } peers { permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44957 } attrs { promote: true } } }
I20260929 21:00:10.296507 7180 sys_catalog.cc:458] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [sys.catalog]: This master's current role is: LEARNER
I20260929 21:00:10.301267 7291 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44957 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:00:10.304327 7295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4b9bcd97294f4ef68830d77516fb1754" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: true } } peers { permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44957 } attrs { promote: true } } }
I20260929 21:00:10.304813 7295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [sys.catalog]: This master's current role is: LEADER
I20260929 21:00:10.305735 7292 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754: attempting to promote NON_VOTER b72fa9367117481cb47b95f56114e583 to VOTER
I20260929 21:00:10.306325 7292 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: false } } peers { permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44957 } attrs { promote: true } }
I20260929 21:00:10.308387 7152 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [term 1 LEARNER]: Refusing update from remote peer 4b9bcd97294f4ef68830d77516fb1754: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 21:00:10.307989 7292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b9bcd97294f4ef68830d77516fb1754" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: true } } peers { permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44957 } attrs { promote: true } } } pending_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: false } } peers { permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44957 } attrs { promote: true } } }
I20260929 21:00:10.308609 7292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [sys.catalog]: This master's current role is: LEADER
I20260929 21:00:10.309094 7295 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 21:00:10.311450 7263 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b72fa9367117481cb47b95f56114e583 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: false } } peers { permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44957 } attrs { promote: true } } }
I20260929 21:00:10.312181 7152 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b72fa9367117481cb47b95f56114e583 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: false } } peers { permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44957 } attrs { promote: true } } }
I20260929 21:00:10.315131 7263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4b9bcd97294f4ef68830d77516fb1754" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: false } } peers { permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44957 } attrs { promote: true } } }
I20260929 21:00:10.315407 7263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [sys.catalog]: This master's current role is: LEADER
I20260929 21:00:10.315668 7263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b9bcd97294f4ef68830d77516fb1754" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: false } } peers { permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44957 } attrs { promote: true } } }
I20260929 21:00:10.315930 7263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [sys.catalog]: This master's current role is: LEADER
I20260929 21:00:10.316134 7180 sys_catalog.cc:455] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b9bcd97294f4ef68830d77516fb1754" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: false } } peers { permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44957 } attrs { promote: true } } }
I20260929 21:00:10.316298 7180 sys_catalog.cc:458] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 21:00:10.327907 7106 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:00:10.328480 7106 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:00:10.328730 7106 tablet_replica.cc:330] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff: stopping tablet replica
I20260929 21:00:10.334874 7264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:10.334433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44957 (local address 127.0.0.1:60444)
0929 21:00:10.334602 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:10.334714 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 21:00:10.335333 6531 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 -> Peer 46eee36a2fe34cd8b2582f89c8f11aff (127.0.97.60:44957): Couldn't send request to peer 46eee36a2fe34cd8b2582f89c8f11aff. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 21:00:10.345934 7106 master.cc:552] Master@127.0.97.60:44957 shutdown complete.
I20260929 21:00:10.346040 7106 master_runner.cc:305] Clearing existing system tablet
I20260929 21:00:10.346637 7106 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:10.346765 7106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:10.348985 7106 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.003s sys 0.001s
I20260929 21:00:10.350404 7299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:10.350695 7106 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:10.350966 7106 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 21:00:10.351068 7106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-2/wal
uuid: "46eee36a2fe34cd8b2582f89c8f11aff"
format_stamp: "Formatted at 2026-09-29 21:00:10 on dist-test-slave-1dms"
server_key: "c2ac0b7704585d7de11e7132569708e2"
server_key_iv: "d499f6a324d244f0d90045b477936441"
server_key_version: "encryptionkey@0"
I20260929 21:00:10.354322 7106 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:00:10.358917 7106 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 21:00:10.359073 7106 log.cc:1196] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 21:00:10.359506 7106 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff: Deleting consensus metadata
I20260929 21:00:10.359753 7106 master_runner.cc:315] Copying system tablet from 127.0.97.62:38557
I20260929 21:00:10.362180 7106 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:38557
I20260929 21:00:10.376823 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:10.362938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60330 (local address 127.0.97.62:38557)
0929 21:00:10.363162 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:10.363172 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:10.363646 (+ 474us) server_negotiation.cc:407] Connection header received
0929 21:00:10.363851 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:10.363857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:10.363933 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:10.364036 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:10.365325 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.366082 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.367297 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.367700 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.367853 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:10.369955 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:10.369961 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:10.369965 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:10.370002 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:10.373713 (+ 3711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:10.374643 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:10.374649 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:10.374652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:10.374828 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:10.375277 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:10.375284 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:10.375288 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:10.375604 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:10.375736 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:10.376237 (+ 501us) server_negotiation.cc:299] Negotiation successful
0929 21:00:10.376453 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 21:00:10.377797 7305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:10.362895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38557 (local address 127.0.0.1:60330)
0929 21:00:10.363536 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:10.363555 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:10.363713 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:10.364064 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:10.364068 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:10.364492 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:10.365179 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:10.365193 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.366227 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.366233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:10.367155 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:10.367165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.368090 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.368096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:10.368223 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:10.368970 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:10.368994 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:10.369773 (+ 779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:10.373890 (+ 4117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:10.373904 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:10.373908 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:10.374505 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:10.374980 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:10.374984 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:10.374988 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:10.375135 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:10.375827 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:10.375832 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:10.376001 (+ 169us) client_negotiation.cc:769] Sending connection context
0929 21:00:10.377086 (+ 1085us) client_negotiation.cc:240] Negotiation successful
0929 21:00:10.377391 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":409,"threads_started":1}
I20260929 21:00:10.378334 6565 tablet_copy_service.cc:140] P 4b9bcd97294f4ef68830d77516fb1754: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 46eee36a2fe34cd8b2582f89c8f11aff ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:60330)
I20260929 21:00:10.378492 6565 tablet_copy_service.cc:161] P 4b9bcd97294f4ef68830d77516fb1754: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 46eee36a2fe34cd8b2582f89c8f11aff at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:60330: session id = 46eee36a2fe34cd8b2582f89c8f11aff-00000000000000000000000000000000
I20260929 21:00:10.382618 6565 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 21:00:10.384400 7106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 21:00:10.391027 7106 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff: tablet copy: Starting download of 0 data blocks...
I20260929 21:00:10.391304 7106 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff: tablet copy: Starting download of 1 WAL segments...
I20260929 21:00:10.393409 7106 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 21:00:10.397037 6565 tablet_copy_service.cc:342] P 4b9bcd97294f4ef68830d77516fb1754: Request end of tablet copy session 46eee36a2fe34cd8b2582f89c8f11aff-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:60330
I20260929 21:00:10.397166 6565 tablet_copy_service.cc:433] P 4b9bcd97294f4ef68830d77516fb1754: ending tablet copy session 46eee36a2fe34cd8b2582f89c8f11aff-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 46eee36a2fe34cd8b2582f89c8f11aff
I20260929 21:00:10.407339 7106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:10.410008 7306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:10.414239 7307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:10.421986 7106 server_base.cc:1034] running on GCE node
W20260929 21:00:10.422233 7309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:10.422659 7106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:10.423274 7106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:10.424464 7106 hybrid_clock.cc:648] HybridClock initialized: now 1790715610424423 us; error 33 us; skew 500 ppm
I20260929 21:00:10.424708 7106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:00:10 dist-test-slave-1dms krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715610, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:00:10.426855 7106 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260929 21:00:10.434749 7106 webserver.cc:460] Webserver started at http://127.0.97.60:40455/ using document root <none> and password file <none>
I20260929 21:00:10.435058 7106 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:10.435173 7106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:10.437517 7106 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260929 21:00:10.439138 7316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:10.439567 7106 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260929 21:00:10.439692 7106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-2/wal
uuid: "46eee36a2fe34cd8b2582f89c8f11aff"
format_stamp: "Formatted at 2026-09-29 21:00:10 on dist-test-slave-1dms"
server_key: "c2ac0b7704585d7de11e7132569708e2"
server_key_iv: "d499f6a324d244f0d90045b477936441"
server_key_version: "encryptionkey@0"
I20260929 21:00:10.439885 7106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:10.449113 7106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:10.452376 7106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:10.463464 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 6606
I20260929 21:00:10.484961 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41637
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:38557
--webserver_port=40345
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:38557,127.0.97.61:44943
--tserver_master_addrs=127.0.97.62:38557,127.0.97.61:44943,127.0.97.60:44957 with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 21:00:10.509442 7106 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:44957
I20260929 21:00:10.509511 7368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:44957 every 8 connection(s)
I20260929 21:00:10.510607 7106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/master-2/data/info.pb
I20260929 21:00:10.515869 7369 sys_catalog.cc:263] Verifying existing consensus state
I20260929 21:00:10.519752 7369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff: Bootstrap starting.
I20260929 21:00:10.544066 7369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 21:00:10.544661 7369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff: Bootstrap complete.
I20260929 21:00:10.545871 7369 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: false } } peers { permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44957 } attrs { promote: true } }
I20260929 21:00:10.546180 7369 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 46eee36a2fe34cd8b2582f89c8f11aff, State: Initialized, Role: LEARNER
I20260929 21:00:10.546443 7369 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: false } } peers { permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44957 } attrs { promote: true } }
I20260929 21:00:10.549335 7372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: false } } peers { permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44957 } attrs { promote: true } } }
I20260929 21:00:10.549655 7369 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:00:10.549890 7372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [sys.catalog]: This master's current role is: LEARNER
I20260929 21:00:10.576012 7383 catalog_manager.cc:1259] Loaded cluster ID: 1d78b66f93774de588144898a42be9a5
I20260929 21:00:10.576123 7383 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff: loading cluster ID for follower catalog manager: success
I20260929 21:00:10.579605 7383 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff: acquiring CA information for follower catalog manager: success
I20260929 21:00:10.581960 7383 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 21:00:10.722769 7264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:10.715869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44957 (local address 127.0.0.1:60448)
0929 21:00:10.716057 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:10.716077 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:10.716215 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:10.716956 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:10.716961 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:10.716991 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:00:10.717200 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:10.717213 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.718597 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.718602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:10.720495 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:10.720511 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.721871 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.721879 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:10.722208 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:10.722243 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 21:00:10.722346 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 21:00:10.722487 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 21:00:10.722869 7384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:10.716018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60448 (local address 127.0.97.60:44957)
0929 21:00:10.716596 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:10.716600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:10.716618 (+ 18us) server_negotiation.cc:407] Connection header received
0929 21:00:10.716686 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:10.716689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:10.716757 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:10.716927 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:00:10.717353 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.718450 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.720699 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.721674 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.721791 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:10.721991 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:10.722397 (+ 406us) server_negotiation.cc:299] Negotiation successful
0929 21:00:10.722565 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":312,"threads_started":1}
I20260929 21:00:10.723915 7343 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 21:00:10.724543 7372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b9bcd97294f4ef68830d77516fb1754. Latest consensus state: current_term: 1 leader_uuid: "4b9bcd97294f4ef68830d77516fb1754" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: false } } peers { permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44957 } attrs { promote: true } } }
I20260929 21:00:10.724915 7372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [sys.catalog]: This master's current role is: LEARNER
W20260929 21:00:10.850498 7355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:10.850761 7355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:10.850906 7355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:10.858286 7355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:10.858390 7355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:10.858563 7355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 21:00:10.868391 7355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=40345
--tserver_master_addrs=127.0.97.62:38557,127.0.97.61:44943,127.0.97.60:44957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:00:10.869851 7355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:10.871111 7355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:10.884503 7389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:10.884449 7390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:10.884528 7392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:10.885428 7355 server_base.cc:1034] running on GCE node
I20260929 21:00:10.886127 7355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:10.887195 7355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:10.888413 7355 hybrid_clock.cc:648] HybridClock initialized: now 1790715610888377 us; error 52 us; skew 500 ppm
I20260929 21:00:10.888727 7355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:00:10 dist-test-slave-1dms krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715610, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:00:10.891901 7355 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260929 21:00:10.893350 7355 webserver.cc:460] Webserver started at http://127.0.97.1:40345/ using document root <none> and password file <none>
I20260929 21:00:10.894021 7355 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:10.894176 7355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:10.899682 7355 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 21:00:10.902381 7399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:10.903785 7355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 21:00:10.903956 7355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-0/wal
uuid: "527d3ead14f6443b9fef7865f31d7227"
format_stamp: "Formatted at 2026-09-29 21:00:07 on dist-test-slave-1dms"
server_key: "17e956282f8d0e3b5f64512941eea73a"
server_key_iv: "3ea27eac97cee66086f6a302bc7087e7"
server_key_version: "encryptionkey@0"
I20260929 21:00:10.904979 7355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:10.923584 7355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:10.927620 7355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:00:10.927902 7355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:10.928808 7355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:00:10.930168 7355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:00:10.930261 7355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:10.930392 7355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:00:10.930467 7355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:10.966118 7355 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41637
I20260929 21:00:10.966207 7511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41637 every 8 connection(s)
I20260929 21:00:10.967559 7355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
I20260929 21:00:10.972041 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 7355
I20260929 21:00:10.972441 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 6740
Sep 29 21:00:10 dist-test-slave-1dms krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715610, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 29 21:00:10 dist-test-slave-1dms krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715610, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 29 21:00:11 dist-test-slave-1dms krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
Sep 29 21:00:11 dist-test-slave-1dms krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715610, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 21:00:11.010706 7384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:10.971976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38635 (local address 127.0.97.60:44957)
0929 21:00:10.972206 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:10.972215 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:10.988645 (+ 16430us) server_negotiation.cc:407] Connection header received
0929 21:00:10.989278 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:10.989288 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:10.989396 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:10.989485 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:10.992772 (+ 3287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.994239 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.996767 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.997462 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.997580 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:11.003050 (+ 5470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:11.003058 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:11.003063 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:11.003106 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:11.006861 (+ 3755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:11.007877 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:11.007881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:11.007884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:11.008026 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:11.008583 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:11.008591 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:11.008600 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:11.008990 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:11.009116 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:11.010135 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0929 21:00:11.010360 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 21:00:11.011897 7517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:10.971348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44957 (local address 127.0.97.1:38635)
0929 21:00:10.988503 (+ 17155us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:10.988530 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:10.988915 (+ 385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:10.990100 (+ 1185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:10.990116 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:10.991386 (+ 1270us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:10.992501 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:10.992532 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.994486 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.994494 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:10.996610 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:10.996621 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.998097 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.998104 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:10.998385 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:10.999198 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:10.999226 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:11.002795 (+ 3569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:11.007017 (+ 4222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:11.007025 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:11.007043 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:11.007743 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:11.008224 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:11.008228 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:11.008232 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:11.008439 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:11.009296 (+ 857us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:11.009303 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:11.009709 (+ 406us) client_negotiation.cc:769] Sending connection context
0929 21:00:11.010061 (+ 352us) client_negotiation.cc:240] Negotiation successful
0929 21:00:11.010561 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9372,"mutex_wait_us":55,"thread_start_us":7798,"threads_started":1}
I20260929 21:00:11.013638 7512 heartbeater.cc:347] Connected to a master server at 127.0.97.60:44957
I20260929 21:00:11.014089 7512 heartbeater.cc:464] Registering TS with master...
I20260929 21:00:11.014613 7520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:10.973577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45524 (local address 127.0.97.2:40605)
0929 21:00:10.985374 (+ 11797us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:10.985386 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:10.991036 (+ 5650us) server_negotiation.cc:407] Connection header received
0929 21:00:10.991249 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:10.991259 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:10.991524 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:10.991692 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:10.994353 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.996089 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.998478 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.999365 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.999456 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:11.006632 (+ 7176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:11.006643 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:11.006657 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:11.006692 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:11.011104 (+ 4412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:11.011938 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:11.011946 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:11.011949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:11.012111 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:11.012562 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:11.012567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:11.012570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:11.013005 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:11.013278 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:11.013959 (+ 681us) server_negotiation.cc:299] Negotiation successful
0929 21:00:11.014228 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11588,"thread_start_us":11417,"threads_started":1}
I20260929 21:00:11.015041 7512 heartbeater.cc:511] Master 127.0.97.60:44957 requested a full tablet report, sending...
I20260929 21:00:11.015425 7516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:10.970696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45725 (local address 127.0.97.61:44943)
0929 21:00:10.989313 (+ 18617us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:10.989322 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:10.989354 (+ 32us) server_negotiation.cc:407] Connection header received
0929 21:00:10.989426 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:10.989435 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:10.989544 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:10.989690 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:10.997174 (+ 7484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.998262 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:11.000153 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:11.000933 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:11.001017 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:11.007137 (+ 6120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:11.007144 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:11.007149 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:11.007196 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:11.011272 (+ 4076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:11.012337 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:11.012342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:11.012345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:11.012482 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:11.013021 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:11.013027 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:11.013032 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:11.013418 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:11.013574 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:11.014397 (+ 823us) server_negotiation.cc:299] Negotiation successful
0929 21:00:11.014962 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18466,"thread_start_us":18352,"threads_started":1}
I20260929 21:00:11.016567 7518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:10.971348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44943 (local address 127.0.97.1:45725)
0929 21:00:10.984279 (+ 12931us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:10.984344 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:10.988523 (+ 4179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:10.990099 (+ 1576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:10.990116 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:10.991614 (+ 1498us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:10.996992 (+ 5378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:10.997004 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.998452 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.998456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:10.999966 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:10.999985 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:11.001883 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:11.001890 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:11.002076 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:11.002672 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:11.002697 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:11.006924 (+ 4227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:11.011417 (+ 4493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:11.011421 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:11.011424 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:11.011938 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:11.012625 (+ 687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:11.012629 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:11.012632 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:11.012890 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:11.013674 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:11.013681 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:11.014047 (+ 366us) client_negotiation.cc:769] Sending connection context
0929 21:00:11.015681 (+ 1634us) client_negotiation.cc:240] Negotiation successful
0929 21:00:11.015945 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12703,"mutex_wait_us":6288,"spinlock_wait_cycles":19840,"thread_start_us":12435,"threads_started":1}
I20260929 21:00:11.016691 6863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:00:11.017529 7514 heartbeater.cc:347] Connected to a master server at 127.0.97.61:44943
I20260929 21:00:11.017695 7514 heartbeater.cc:464] Registering TS with master...
I20260929 21:00:11.017877 7333 ts_manager.cc:194] Registered new tserver with Master: 527d3ead14f6443b9fef7865f31d7227 (127.0.97.1:41637)
I20260929 21:00:11.018182 7514 heartbeater.cc:511] Master 127.0.97.61:44943 requested a full tablet report, sending...
I20260929 21:00:11.019635 7142 ts_manager.cc:194] Registered new tserver with Master: 527d3ead14f6443b9fef7865f31d7227 (127.0.97.1:41637)
I20260929 21:00:11.022444 7521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:10.981137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45735 (local address 127.0.97.62:38557)
0929 21:00:10.995635 (+ 14498us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:10.995641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:10.996204 (+ 563us) server_negotiation.cc:407] Connection header received
0929 21:00:10.996459 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:10.996467 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:10.996556 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:10.996639 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:10.998287 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.999721 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:11.001812 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:11.002544 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:11.002667 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:11.007944 (+ 5277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:11.007953 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:11.007960 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:11.007996 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:11.013636 (+ 5640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:11.014622 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:11.014630 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:11.014633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:11.014788 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:11.015251 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:11.015258 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:11.015261 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:11.019070 (+ 3809us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:11.019185 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:11.021518 (+ 2333us) server_negotiation.cc:299] Negotiation successful
0929 21:00:11.021881 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14366,"thread_start_us":14505,"threads_started":1}
I20260929 21:00:11.024247 7522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:10.981162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38557 (local address 127.0.97.1:45735)
0929 21:00:10.996108 (+ 14946us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:10.996134 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:10.996304 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:10.996890 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:10.996894 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:10.997405 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:10.998117 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:10.998129 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:10.999885 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:10.999890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:11.001629 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:11.001647 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:11.003035 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:11.003043 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:11.003263 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:11.004151 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:11.004176 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:11.007721 (+ 3545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:11.013883 (+ 6162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:11.013890 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:11.013895 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:11.014489 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:11.014941 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:11.014945 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:11.014948 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:11.015114 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:11.020971 (+ 5857us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:11.020979 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:11.021161 (+ 182us) client_negotiation.cc:769] Sending connection context
0929 21:00:11.023184 (+ 2023us) client_negotiation.cc:240] Negotiation successful
0929 21:00:11.023617 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14830,"thread_start_us":14688,"threads_started":1}
I20260929 21:00:11.025283 7513 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38557
I20260929 21:00:11.025416 7513 heartbeater.cc:464] Registering TS with master...
I20260929 21:00:11.025781 7513 heartbeater.cc:511] Master 127.0.97.62:38557 requested a full tablet report, sending...
I20260929 21:00:11.026891 6545 ts_manager.cc:194] Re-registered known tserver with Master: 527d3ead14f6443b9fef7865f31d7227 (127.0.97.1:41637)
I20260929 21:00:11.028007 6545 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45735
I20260929 21:00:11.103682 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 6740
I20260929 21:00:11.120394 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:40605
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:38557
--webserver_port=37581
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:38557,127.0.97.61:44943
--tserver_master_addrs=127.0.97.62:38557,127.0.97.61:44943,127.0.97.60:44957 with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 21:00:11.328372 7295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b9bcd97294f4ef68830d77516fb1754" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: false } } peers { permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44957 } attrs { promote: true } } }
I20260929 21:00:11.328805 7295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [sys.catalog]: This master's current role is: LEADER
I20260929 21:00:11.330024 7525 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754: attempting to promote NON_VOTER 46eee36a2fe34cd8b2582f89c8f11aff to VOTER
I20260929 21:00:11.330566 7525 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: false } } peers { permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44957 } attrs { promote: false } }
I20260929 21:00:11.332482 7343 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [term 1 LEARNER]: Refusing update from remote peer 4b9bcd97294f4ef68830d77516fb1754: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 21:00:11.332527 7152 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [term 1 FOLLOWER]: Refusing update from remote peer 4b9bcd97294f4ef68830d77516fb1754: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 21:00:11.333055 7526 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44957 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 21:00:11.333446 7295 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 21:00:11.336421 7526 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 46eee36a2fe34cd8b2582f89c8f11aff (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: false } } peers { permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44957 } attrs { promote: false } } }
I20260929 21:00:11.336962 7152 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 46eee36a2fe34cd8b2582f89c8f11aff (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: false } } peers { permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44957 } attrs { promote: false } } }
I20260929 21:00:11.341845 7525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4b9bcd97294f4ef68830d77516fb1754" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: false } } peers { permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44957 } attrs { promote: false } } }
I20260929 21:00:11.341917 7295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b9bcd97294f4ef68830d77516fb1754" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: false } } peers { permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44957 } attrs { promote: false } } }
I20260929 21:00:11.342152 7525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [sys.catalog]: This master's current role is: LEADER
I20260929 21:00:11.342229 7295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [sys.catalog]: This master's current role is: LEADER
I20260929 21:00:11.344738 7531 sys_catalog.cc:455] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b9bcd97294f4ef68830d77516fb1754" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: false } } peers { permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44957 } attrs { promote: false } } }
I20260929 21:00:11.345136 7531 sys_catalog.cc:458] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 21:00:11.345554 7526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b9bcd97294f4ef68830d77516fb1754" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: false } } peers { permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44957 } attrs { promote: false } } }
I20260929 21:00:11.345836 7526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754 [sys.catalog]: This master's current role is: LEADER
I20260929 21:00:11.346169 7343 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 46eee36a2fe34cd8b2582f89c8f11aff (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: false } } peers { permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44957 } attrs { promote: false } } }
I20260929 21:00:11.351181 7532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b9bcd97294f4ef68830d77516fb1754" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: false } } peers { permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44957 } attrs { promote: false } } }
I20260929 21:00:11.351506 7532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [sys.catalog]: This master's current role is: FOLLOWER
W20260929 21:00:11.455354 7524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:11.455772 7524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:11.455935 7524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:11.463474 7524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:11.463574 7524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:11.463761 7524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 21:00:11.473618 7524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:40605
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=37581
--tserver_master_addrs=127.0.97.62:38557,127.0.97.61:44943,127.0.97.60:44957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:00:11.474980 7524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:11.476349 7524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:11.489435 7538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:11.489676 7540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:11.489955 7537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:11.490413 7524 server_base.cc:1034] running on GCE node
I20260929 21:00:11.490974 7524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:11.491842 7524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:11.493053 7524 hybrid_clock.cc:648] HybridClock initialized: now 1790715611493019 us; error 52 us; skew 500 ppm
I20260929 21:00:11.493352 7524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:00:11 dist-test-slave-1dms krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715611, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:00:11.496389 7524 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260929 21:00:11.497843 7524 webserver.cc:460] Webserver started at http://127.0.97.2:37581/ using document root <none> and password file <none>
I20260929 21:00:11.498638 7524 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:11.498785 7524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:11.505254 7524 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260929 21:00:11.508510 7547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:11.510116 7524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 21:00:11.510258 7524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-1/wal
uuid: "c5523b2f4ad14d37ba187ff45359077e"
format_stamp: "Formatted at 2026-09-29 21:00:08 on dist-test-slave-1dms"
server_key: "268434c6bfd56f6ca2b7f4fa667375e7"
server_key_iv: "8a2cf75c9bd5de13d380f97e5b542b03"
server_key_version: "encryptionkey@0"
I20260929 21:00:11.511499 7524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:11.534061 7524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:11.538326 7524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:00:11.538538 7524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:11.539469 7524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:00:11.540964 7524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:00:11.541074 7524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:11.541173 7524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:00:11.541230 7524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:11.573537 7524 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40605
I20260929 21:00:11.573626 7659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40605 every 8 connection(s)
I20260929 21:00:11.574944 7524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
I20260929 21:00:11.580137 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 7524
I20260929 21:00:11.580473 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 6873
Sep 29 21:00:11 dist-test-slave-1dms krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715611, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 29 21:00:11 dist-test-slave-1dms krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715611, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 29 21:00:11 dist-test-slave-1dms krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715611, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 29 21:00:11 dist-test-slave-1dms krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260929 21:00:11.613380 7665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:11.577741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60185 (local address 127.0.97.61:44943)
0929 21:00:11.578371 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:11.578375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:11.579520 (+ 1145us) server_negotiation.cc:407] Connection header received
0929 21:00:11.590153 (+ 10633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:11.590162 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:11.590307 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:11.590429 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:11.595713 (+ 5284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:11.596986 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:11.599144 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:11.599753 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:11.600135 (+ 382us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:11.604476 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:11.604485 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:11.604491 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:11.604543 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:11.608884 (+ 4341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:11.610369 (+ 1485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:11.610375 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:11.610379 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:11.610524 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:11.611059 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:11.611065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:11.611068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:11.611410 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:11.611556 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:11.612715 (+ 1159us) server_negotiation.cc:299] Negotiation successful
0929 21:00:11.612975 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":436,"threads_started":1}
I20260929 21:00:11.613881 7669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:11.580130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51579 (local address 127.0.97.60:44957)
0929 21:00:11.594434 (+ 14304us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:11.594442 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:11.595364 (+ 922us) server_negotiation.cc:407] Connection header received
0929 21:00:11.595506 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:11.595512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:11.595600 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:11.595706 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:11.596934 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:11.597982 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:11.599862 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:11.600475 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:11.600601 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:11.605157 (+ 4556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:11.605163 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:11.605167 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:11.605205 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:11.609221 (+ 4016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:11.610846 (+ 1625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:11.610853 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:11.610860 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:11.610996 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:11.611444 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:11.611451 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:11.611457 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:11.611802 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:11.611943 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:11.612975 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0929 21:00:11.613177 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14184,"thread_start_us":14088,"threads_started":1}
I20260929 21:00:11.615001 7664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:11.577761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38851 (local address 127.0.97.62:38557)
0929 21:00:11.578222 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:11.578228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:11.579449 (+ 1221us) server_negotiation.cc:407] Connection header received
0929 21:00:11.594467 (+ 15018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:11.594477 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:11.594588 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:11.594698 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:11.596312 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:11.597410 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:11.601001 (+ 3591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:11.601632 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:11.601741 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:11.607621 (+ 5880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:11.607630 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:11.607637 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:11.607686 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:11.611910 (+ 4224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:11.612610 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:11.612616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:11.612618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:11.612803 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:11.613285 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:11.613290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:11.613294 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:11.613604 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:11.613722 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:11.614408 (+ 686us) server_negotiation.cc:299] Negotiation successful
0929 21:00:11.614627 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":219,"threads_started":1}
I20260929 21:00:11.615731 7668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:11.579934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44957 (local address 127.0.97.2:51579)
0929 21:00:11.595240 (+ 15306us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:11.595259 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:11.595384 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:11.595905 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:11.595909 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:11.596473 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:11.596732 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:11.596743 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:11.598130 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:11.598135 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:11.599682 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:11.599696 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:11.601006 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:11.601015 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:11.601203 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:11.601811 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:11.601832 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:11.604950 (+ 3118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:11.609384 (+ 4434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:11.609389 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:11.609393 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:11.610704 (+ 1311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:11.611127 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:11.611132 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:11.611135 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:11.611304 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:11.612004 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:11.612011 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:11.612154 (+ 143us) client_negotiation.cc:769] Sending connection context
0929 21:00:11.612420 (+ 266us) client_negotiation.cc:240] Negotiation successful
0929 21:00:11.614202 (+ 1782us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14806,"mutex_wait_us":3,"thread_start_us":14699,"threads_started":1}
I20260929 21:00:11.615759 7667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:11.578566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44943 (local address 127.0.97.2:60185)
0929 21:00:11.579314 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:11.579346 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:11.589831 (+ 10485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:11.590975 (+ 1144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:11.590992 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:11.592207 (+ 1215us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:11.593314 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:11.593346 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:11.597255 (+ 3909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:11.597260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:11.599003 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:11.599014 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:11.600004 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:11.600010 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:11.600299 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:11.601120 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:11.601156 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:11.604214 (+ 3058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:11.609094 (+ 4880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:11.609107 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:11.609126 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:11.610212 (+ 1086us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:11.610661 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:11.610667 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:11.610672 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:11.610910 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:11.611636 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:11.611645 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:11.612076 (+ 431us) client_negotiation.cc:769] Sending connection context
0929 21:00:11.612420 (+ 344us) client_negotiation.cc:240] Negotiation successful
0929 21:00:11.614386 (+ 1966us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":1620,"spinlock_wait_cycles":55040,"thread_start_us":453,"threads_started":1}
I20260929 21:00:11.616487 7666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:11.578368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38557 (local address 127.0.97.2:38851)
0929 21:00:11.579301 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:11.579345 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:11.589476 (+ 10131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:11.595291 (+ 5815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:11.595295 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:11.595814 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:11.596111 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:11.596123 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:11.597584 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:11.597589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:11.600832 (+ 3243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:11.600846 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:11.602382 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:11.602388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:11.602571 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:11.604321 (+ 1750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:11.604342 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:11.607435 (+ 3093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:11.612058 (+ 4623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:11.612062 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:11.612066 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:11.612492 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:11.612967 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:11.612971 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:11.612974 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:11.613163 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:11.613820 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:11.613846 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:11.614079 (+ 233us) client_negotiation.cc:769] Sending connection context
0929 21:00:11.615251 (+ 1172us) client_negotiation.cc:240] Negotiation successful
0929 21:00:11.615509 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"mutex_wait_us":1,"thread_start_us":418,"threads_started":1}
I20260929 21:00:11.617290 7662 heartbeater.cc:347] Connected to a master server at 127.0.97.61:44943
I20260929 21:00:11.617393 7660 heartbeater.cc:347] Connected to a master server at 127.0.97.60:44957
I20260929 21:00:11.617413 7671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:11.581621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43770 (local address 127.0.97.3:36861)
0929 21:00:11.595339 (+ 13718us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:11.595348 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:11.595379 (+ 31us) server_negotiation.cc:407] Connection header received
0929 21:00:11.595454 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:11.595462 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:11.595716 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:11.597891 (+ 2175us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:11.598146 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:11.599726 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:11.601545 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:11.602812 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:11.602993 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:11.609561 (+ 6568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:11.609573 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:11.609592 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:11.609634 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:11.613474 (+ 3840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:11.614303 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:11.614313 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:11.614320 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:11.614466 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:11.614902 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:11.614907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:11.614909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:11.615279 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:11.615549 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:11.616712 (+ 1163us) server_negotiation.cc:299] Negotiation successful
0929 21:00:11.616977 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13533,"thread_start_us":13849,"threads_started":1}
I20260929 21:00:11.617604 7662 heartbeater.cc:464] Registering TS with master...
I20260929 21:00:11.617607 7660 heartbeater.cc:464] Registering TS with master...
I20260929 21:00:11.618222 7661 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38557
I20260929 21:00:11.618350 7661 heartbeater.cc:464] Registering TS with master...
I20260929 21:00:11.618599 7660 heartbeater.cc:511] Master 127.0.97.60:44957 requested a full tablet report, sending...
I20260929 21:00:11.618737 6996 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:00:11.618831 7661 heartbeater.cc:511] Master 127.0.97.62:38557 requested a full tablet report, sending...
I20260929 21:00:11.619464 7662 heartbeater.cc:511] Master 127.0.97.61:44943 requested a full tablet report, sending...
I20260929 21:00:11.620391 7333 ts_manager.cc:194] Registered new tserver with Master: c5523b2f4ad14d37ba187ff45359077e (127.0.97.2:40605)
I20260929 21:00:11.620502 6545 ts_manager.cc:194] Re-registered known tserver with Master: c5523b2f4ad14d37ba187ff45359077e (127.0.97.2:40605)
I20260929 21:00:11.621464 7142 ts_manager.cc:194] Registered new tserver with Master: c5523b2f4ad14d37ba187ff45359077e (127.0.97.2:40605)
I20260929 21:00:11.622030 6545 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38851
I20260929 21:00:11.704993 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 6873
I20260929 21:00:11.722415 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:36861
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:38557
--webserver_port=39349
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:38557,127.0.97.61:44943
--tserver_master_addrs=127.0.97.62:38557,127.0.97.61:44943,127.0.97.60:44957 with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 21:00:12.030287 7513 heartbeater.cc:503] Master 127.0.97.62:38557 was elected leader, sending a full tablet report...
W20260929 21:00:12.044693 7673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:12.045018 7673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:12.045212 7673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:12.052263 7673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:12.052377 7673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:12.052539 7673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 21:00:12.061908 7673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:36861
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=39349
--tserver_master_addrs=127.0.97.62:38557,127.0.97.61:44943,127.0.97.60:44957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:00:12.063133 7673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:12.064513 7673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:12.077265 7679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:12.077379 7680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:12.077351 7682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:12.078123 7673 server_base.cc:1034] running on GCE node
I20260929 21:00:12.078685 7673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:12.079551 7673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:12.080761 7673 hybrid_clock.cc:648] HybridClock initialized: now 1790715612080701 us; error 43 us; skew 500 ppm
I20260929 21:00:12.081096 7673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:00:12 dist-test-slave-1dms krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715612, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:00:12.084048 7673 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260929 21:00:12.085551 7673 webserver.cc:460] Webserver started at http://127.0.97.3:39349/ using document root <none> and password file <none>
I20260929 21:00:12.086248 7673 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:12.086393 7673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:12.091405 7673 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 21:00:12.093930 7689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:12.095081 7673 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 21:00:12.095243 7673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-2/wal
uuid: "34844c02e0f34d5fab45f43e4e5b9956"
format_stamp: "Formatted at 2026-09-29 21:00:08 on dist-test-slave-1dms"
server_key: "e671cc35f9409d3345e8183fae290d0c"
server_key_iv: "97811185d5c81081f530692bc9a034c5"
server_key_version: "encryptionkey@0"
I20260929 21:00:12.096156 7673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:12.108311 7673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:12.112449 7673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:00:12.112706 7673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:12.113660 7673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:00:12.115125 7673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:00:12.115234 7673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:12.115357 7673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:00:12.115453 7673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:12.147388 7673 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36861
I20260929 21:00:12.147469 7801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36861 every 8 connection(s)
I20260929 21:00:12.148782 7673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
I20260929 21:00:12.151866 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 7673
Sep 29 21:00:12 dist-test-slave-1dms krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715612, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 21:00:12.176771 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:12.151608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41279 (local address 127.0.97.62:38557)
0929 21:00:12.152037 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:12.152044 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:12.153235 (+ 1191us) server_negotiation.cc:407] Connection header received
0929 21:00:12.156252 (+ 3017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:12.156262 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:12.156365 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:12.156457 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:12.159034 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.160389 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.163275 (+ 2886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.164075 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.164216 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:12.168811 (+ 4595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:12.168821 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:12.168829 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:12.168914 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:12.173478 (+ 4564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:12.174434 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:12.174439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:12.174443 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:12.174591 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:12.174992 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:12.175000 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:12.175006 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:12.175392 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:12.175554 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:12.176181 (+ 627us) server_negotiation.cc:299] Negotiation successful
0929 21:00:12.176388 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":383,"threads_started":1}
I20260929 21:00:12.177866 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:12.152202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38557 (local address 127.0.97.3:41279)
0929 21:00:12.153077 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:12.153135 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:12.155901 (+ 2766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:12.157084 (+ 1183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:12.157098 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:12.157948 (+ 850us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:12.158818 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:12.158846 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.160602 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.160609 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:12.163093 (+ 2484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:12.163111 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.164325 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.164336 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:12.164729 (+ 393us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:12.165647 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:12.165679 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:12.168555 (+ 2876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:12.173904 (+ 5349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:12.173911 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:12.173923 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:12.174307 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:12.174695 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:12.174701 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:12.174705 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:12.174873 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:12.175518 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:12.175525 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:12.175844 (+ 319us) client_negotiation.cc:769] Sending connection context
0929 21:00:12.176095 (+ 251us) client_negotiation.cc:240] Negotiation successful
0929 21:00:12.176591 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":712,"threads_started":1}
I20260929 21:00:12.179468 7803 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38557
I20260929 21:00:12.179801 7803 heartbeater.cc:464] Registering TS with master...
I20260929 21:00:12.180855 7803 heartbeater.cc:511] Master 127.0.97.62:38557 requested a full tablet report, sending...
I20260929 21:00:12.182766 6545 ts_manager.cc:194] Re-registered known tserver with Master: 34844c02e0f34d5fab45f43e4e5b9956 (127.0.97.3:36861)
I20260929 21:00:12.184234 6545 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41279
Sep 29 21:00:12 dist-test-slave-1dms krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715612, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 29 21:00:12 dist-test-slave-1dms krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715612, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260929 21:00:12.212704 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:12.167790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60342 (local address 127.0.97.62:38557)
0929 21:00:12.177091 (+ 9301us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:12.177098 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:12.177114 (+ 16us) server_negotiation.cc:407] Connection header received
0929 21:00:12.177158 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:12.177163 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:12.177238 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:12.177333 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:12.180927 (+ 3594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.185083 (+ 4156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.195281 (+ 10198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.196155 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.196309 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:12.202142 (+ 5833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:12.202152 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:12.202160 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:12.202206 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:12.207190 (+ 4984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:12.209492 (+ 2302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:12.209498 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:12.209501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:12.209646 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:12.211132 (+ 1486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:12.211138 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:12.211142 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:12.211517 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:12.211764 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:12.212069 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 21:00:12.212278 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9204,"thread_start_us":37039,"threads_started":1}
I20260929 21:00:12.212913 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:12.153928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44957 (local address 127.0.97.3:35913)
0929 21:00:12.178265 (+ 24337us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:12.178281 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:12.178399 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:12.179649 (+ 1250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:12.179657 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:12.180361 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:12.180631 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:12.180646 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.187018 (+ 6372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.187031 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:12.189374 (+ 2343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:12.189394 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.191362 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.191370 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:12.191643 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:12.192528 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:12.192559 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:12.199860 (+ 7301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:12.209563 (+ 9703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:12.209571 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:12.209576 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:12.209982 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:12.210949 (+ 967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:12.210955 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:12.210958 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:12.211111 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:12.211802 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:12.211810 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:12.211968 (+ 158us) client_negotiation.cc:769] Sending connection context
0929 21:00:12.212101 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 21:00:12.212418 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24258,"mutex_wait_us":38,"thread_start_us":11590,"threads_started":1}
I20260929 21:00:12.213896 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:12.154251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35913 (local address 127.0.97.60:44957)
0929 21:00:12.165238 (+ 10987us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:12.165245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:12.178365 (+ 13120us) server_negotiation.cc:407] Connection header received
0929 21:00:12.179075 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:12.179084 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:12.179245 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:12.179364 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:12.185248 (+ 5884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.186805 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.189583 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.190958 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.191113 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:12.200097 (+ 8984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:12.200107 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:12.200115 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:12.200169 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:12.209413 (+ 9244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:12.210111 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:12.210119 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:12.210122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:12.210683 (+ 561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:12.211252 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:12.211257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:12.211261 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:12.211601 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:12.211743 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:12.213261 (+ 1518us) server_negotiation.cc:299] Negotiation successful
0929 21:00:12.213534 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10867,"thread_start_us":10772,"threads_started":1}
I20260929 21:00:12.214895 7802 heartbeater.cc:347] Connected to a master server at 127.0.97.60:44957
I20260929 21:00:12.215041 7802 heartbeater.cc:464] Registering TS with master...
I20260929 21:00:12.215477 7802 heartbeater.cc:511] Master 127.0.97.60:44957 requested a full tablet report, sending...
I20260929 21:00:12.216239 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:12.151888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57477 (local address 127.0.97.61:44943)
0929 21:00:12.163047 (+ 11159us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:12.163056 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:12.165750 (+ 2694us) server_negotiation.cc:407] Connection header received
0929 21:00:12.166046 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:12.166055 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:12.166154 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:12.166248 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:12.190084 (+ 23836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.191331 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.202478 (+ 11147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.203194 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.203321 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:12.207845 (+ 4524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:12.207852 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:12.207856 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:12.207893 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:12.212128 (+ 4235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:12.213511 (+ 1383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:12.213516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:12.213519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:12.213661 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:12.214085 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:12.214090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:12.214093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:12.214387 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:12.214551 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:12.215631 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0929 21:00:12.215841 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10997,"thread_start_us":11172,"threads_started":1}
I20260929 21:00:12.216692 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:12.152788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44943 (local address 127.0.97.3:57477)
0929 21:00:12.165644 (+ 12856us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:12.165663 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:12.165885 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:12.166502 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:12.166517 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:12.167298 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:12.187880 (+ 20582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:12.187900 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.191586 (+ 3686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.191592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:12.198842 (+ 7250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:12.198862 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.203605 (+ 4743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.203612 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:12.203795 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:12.204647 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:12.204674 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:12.207645 (+ 2971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:12.212253 (+ 4608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:12.212257 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:12.212261 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:12.213153 (+ 892us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:12.213811 (+ 658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:12.213816 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:12.213819 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:12.213955 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:12.214642 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:12.214650 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:12.214819 (+ 169us) client_negotiation.cc:769] Sending connection context
0929 21:00:12.214965 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 21:00:12.215263 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12715,"mutex_wait_us":4750,"thread_start_us":40395,"threads_started":1}
I20260929 21:00:12.216773 7333 ts_manager.cc:194] Registered new tserver with Master: 34844c02e0f34d5fab45f43e4e5b9956 (127.0.97.3:36861)
I20260929 21:00:12.217950 7804 heartbeater.cc:347] Connected to a master server at 127.0.97.61:44943
I20260929 21:00:12.218088 7804 heartbeater.cc:464] Registering TS with master...
I20260929 21:00:12.218448 7804 heartbeater.cc:511] Master 127.0.97.61:44943 requested a full tablet report, sending...
I20260929 21:00:12.219512 7142 ts_manager.cc:194] Registered new tserver with Master: 34844c02e0f34d5fab45f43e4e5b9956 (127.0.97.3:36861)
I20260929 21:00:12.222700 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:12.185326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60460 (local address 127.0.97.60:44957)
0929 21:00:12.201379 (+ 16053us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:12.201384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:12.202596 (+ 1212us) server_negotiation.cc:407] Connection header received
0929 21:00:12.202816 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:12.202821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:12.202894 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:12.202967 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:12.205496 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.206772 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.209552 (+ 2780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.210237 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.210360 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:12.215518 (+ 5158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:12.215524 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:12.215528 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:12.215564 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:12.219603 (+ 4039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:12.220359 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:12.220364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:12.220367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:12.220494 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:12.221271 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:12.221276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:12.221281 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:12.221583 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:12.221843 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:12.222128 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 21:00:12.222343 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15928,"thread_start_us":16055,"threads_started":1}
I20260929 21:00:12.222967 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:12.173691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36592 (local address 127.0.97.61:44943)
0929 21:00:12.197041 (+ 23350us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:12.197049 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:12.197069 (+ 20us) server_negotiation.cc:407] Connection header received
0929 21:00:12.197138 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:12.197146 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:12.197231 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:12.199620 (+ 2389us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:12.199910 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.201563 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.203986 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.204720 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.205634 (+ 914us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:12.208219 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:12.208227 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:12.208233 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:12.208271 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:12.220101 (+ 11830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:12.220884 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:12.220890 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:12.220895 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:12.221060 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:12.221488 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:12.221494 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:12.221497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:12.221808 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:12.222088 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:12.222380 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 21:00:12.222586 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23171,"thread_start_us":23630,"threads_started":1}
I20260929 21:00:12.218990 6545 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 21:00:12.228106 6545 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 21:00:12.274729 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:12.247079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45536 (local address 127.0.97.2:40605)
0929 21:00:12.268089 (+ 21010us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:12.268099 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:12.268128 (+ 29us) server_negotiation.cc:407] Connection header received
0929 21:00:12.268229 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:12.268236 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:12.268477 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:12.268670 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:00:12.269344 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.270282 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.272556 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.273759 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.273897 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:12.274104 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:12.274303 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 21:00:12.274451 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20862,"thread_start_us":20756,"threads_started":1}
I20260929 21:00:12.274807 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:12.246863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40605 (local address 127.0.0.1:45536)
0929 21:00:12.247463 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:12.247482 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:12.247595 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:12.268679 (+ 21084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:12.268684 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:12.268709 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:00:12.269151 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:12.269166 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.270414 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.270422 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:12.272363 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:12.272376 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.273929 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.273938 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:12.274095 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:12.274133 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 21:00:12.274250 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 21:00:12.274379 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":1,"thread_start_us":297,"threads_started":1}
I20260929 21:00:12.275190 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:12.247495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41637 (local address 127.0.0.1:46856)
0929 21:00:12.265982 (+ 18487us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:12.266006 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:12.266591 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:12.267693 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:12.267701 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:12.267733 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:00:12.268129 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:12.268141 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.269996 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.270001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:12.272156 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:12.272172 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.273984 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.273990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:12.274158 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:12.274191 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 21:00:12.274262 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 21:00:12.274399 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18326,"mutex_wait_us":7,"thread_start_us":19362,"threads_started":1}
I20260929 21:00:12.275447 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:12.247718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46856 (local address 127.0.97.1:41637)
0929 21:00:12.265417 (+ 17699us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:12.265431 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:12.266563 (+ 1132us) server_negotiation.cc:407] Connection header received
0929 21:00:12.266799 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:12.266812 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:12.267135 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:12.267406 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:00:12.268348 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.269843 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.272386 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.273779 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.274719 (+ 940us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:12.274918 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:12.274979 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 21:00:12.275140 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17455,"thread_start_us":17317,"threads_started":1}
I20260929 21:00:12.276640 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:12.248205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43776 (local address 127.0.97.3:36861)
0929 21:00:12.262349 (+ 14144us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:12.262359 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:12.268205 (+ 5846us) server_negotiation.cc:407] Connection header received
0929 21:00:12.268298 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:12.268303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:12.268476 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:12.268666 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:00:12.269461 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.271072 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.273439 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.274800 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.274930 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:12.275122 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:12.275871 (+ 749us) server_negotiation.cc:299] Negotiation successful
0929 21:00:12.276011 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13959,"thread_start_us":13788,"threads_started":1}
I20260929 21:00:12.276659 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:12.247507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36861 (local address 127.0.0.1:43776)
0929 21:00:12.267088 (+ 19581us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:12.267105 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:12.267212 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:12.268905 (+ 1693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:12.268912 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:12.268936 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:00:12.269241 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:12.269248 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.271213 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.271218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:12.273281 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:12.273294 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.275518 (+ 2224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.275524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:12.275675 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:12.275704 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 21:00:12.275791 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 21:00:12.275894 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19480,"mutex_wait_us":1,"thread_start_us":18151,"threads_started":1}
I20260929 21:00:12.279731 7595 tablet_service.cc:1467] Processing CreateTablet for tablet ad9077bf2f0348f5847171d3ca5d5ba3 (DEFAULT_TABLE table=test-table [id=f8938ae7ce98473ab4c3917bb23f95c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:00:12.279757 7447 tablet_service.cc:1467] Processing CreateTablet for tablet ad9077bf2f0348f5847171d3ca5d5ba3 (DEFAULT_TABLE table=test-table [id=f8938ae7ce98473ab4c3917bb23f95c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:00:12.280110 7737 tablet_service.cc:1467] Processing CreateTablet for tablet ad9077bf2f0348f5847171d3ca5d5ba3 (DEFAULT_TABLE table=test-table [id=f8938ae7ce98473ab4c3917bb23f95c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:00:12.280908 7595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad9077bf2f0348f5847171d3ca5d5ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:00:12.281082 7447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad9077bf2f0348f5847171d3ca5d5ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:00:12.281327 7737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad9077bf2f0348f5847171d3ca5d5ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:00:12.293501 7836 tablet_bootstrap.cc:492] T ad9077bf2f0348f5847171d3ca5d5ba3 P c5523b2f4ad14d37ba187ff45359077e: Bootstrap starting.
I20260929 21:00:12.296042 7837 tablet_bootstrap.cc:492] T ad9077bf2f0348f5847171d3ca5d5ba3 P 527d3ead14f6443b9fef7865f31d7227: Bootstrap starting.
I20260929 21:00:12.297340 7836 tablet_bootstrap.cc:654] T ad9077bf2f0348f5847171d3ca5d5ba3 P c5523b2f4ad14d37ba187ff45359077e: Neither blocks nor log segments found. Creating new log.
I20260929 21:00:12.297384 7838 tablet_bootstrap.cc:492] T ad9077bf2f0348f5847171d3ca5d5ba3 P 34844c02e0f34d5fab45f43e4e5b9956: Bootstrap starting.
I20260929 21:00:12.298664 7836 log.cc:824] T ad9077bf2f0348f5847171d3ca5d5ba3 P c5523b2f4ad14d37ba187ff45359077e: Log is configured to *not* fsync() on all Append() calls
I20260929 21:00:12.300942 7838 tablet_bootstrap.cc:654] T ad9077bf2f0348f5847171d3ca5d5ba3 P 34844c02e0f34d5fab45f43e4e5b9956: Neither blocks nor log segments found. Creating new log.
I20260929 21:00:12.301826 7836 tablet_bootstrap.cc:492] T ad9077bf2f0348f5847171d3ca5d5ba3 P c5523b2f4ad14d37ba187ff45359077e: No bootstrap required, opened a new log
I20260929 21:00:12.302105 7836 ts_tablet_manager.cc:1397] T ad9077bf2f0348f5847171d3ca5d5ba3 P c5523b2f4ad14d37ba187ff45359077e: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 21:00:12.302160 7838 log.cc:824] T ad9077bf2f0348f5847171d3ca5d5ba3 P 34844c02e0f34d5fab45f43e4e5b9956: Log is configured to *not* fsync() on all Append() calls
I20260929 21:00:12.302723 7837 tablet_bootstrap.cc:654] T ad9077bf2f0348f5847171d3ca5d5ba3 P 527d3ead14f6443b9fef7865f31d7227: Neither blocks nor log segments found. Creating new log.
I20260929 21:00:12.303879 7837 log.cc:824] T ad9077bf2f0348f5847171d3ca5d5ba3 P 527d3ead14f6443b9fef7865f31d7227: Log is configured to *not* fsync() on all Append() calls
I20260929 21:00:12.305823 7838 tablet_bootstrap.cc:492] T ad9077bf2f0348f5847171d3ca5d5ba3 P 34844c02e0f34d5fab45f43e4e5b9956: No bootstrap required, opened a new log
I20260929 21:00:12.306084 7838 ts_tablet_manager.cc:1397] T ad9077bf2f0348f5847171d3ca5d5ba3 P 34844c02e0f34d5fab45f43e4e5b9956: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260929 21:00:12.306764 7837 tablet_bootstrap.cc:492] T ad9077bf2f0348f5847171d3ca5d5ba3 P 527d3ead14f6443b9fef7865f31d7227: No bootstrap required, opened a new log
I20260929 21:00:12.307010 7837 ts_tablet_manager.cc:1397] T ad9077bf2f0348f5847171d3ca5d5ba3 P 527d3ead14f6443b9fef7865f31d7227: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260929 21:00:12.311120 7836 raft_consensus.cc:348] T ad9077bf2f0348f5847171d3ca5d5ba3 P c5523b2f4ad14d37ba187ff45359077e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5523b2f4ad14d37ba187ff45359077e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40605 } } peers { permanent_uuid: "527d3ead14f6443b9fef7865f31d7227" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41637 } } peers { permanent_uuid: "34844c02e0f34d5fab45f43e4e5b9956" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36861 } }
I20260929 21:00:12.311477 7836 raft_consensus.cc:374] T ad9077bf2f0348f5847171d3ca5d5ba3 P c5523b2f4ad14d37ba187ff45359077e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:00:12.311571 7836 raft_consensus.cc:729] T ad9077bf2f0348f5847171d3ca5d5ba3 P c5523b2f4ad14d37ba187ff45359077e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5523b2f4ad14d37ba187ff45359077e, State: Initialized, Role: FOLLOWER
I20260929 21:00:12.312218 7836 consensus_queue.cc:260] T ad9077bf2f0348f5847171d3ca5d5ba3 P c5523b2f4ad14d37ba187ff45359077e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5523b2f4ad14d37ba187ff45359077e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40605 } } peers { permanent_uuid: "527d3ead14f6443b9fef7865f31d7227" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41637 } } peers { permanent_uuid: "34844c02e0f34d5fab45f43e4e5b9956" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36861 } }
I20260929 21:00:12.313282 7661 heartbeater.cc:503] Master 127.0.97.62:38557 was elected leader, sending a full tablet report...
I20260929 21:00:12.314002 7836 ts_tablet_manager.cc:1428] T ad9077bf2f0348f5847171d3ca5d5ba3 P c5523b2f4ad14d37ba187ff45359077e: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260929 21:00:12.317082 7838 raft_consensus.cc:348] T ad9077bf2f0348f5847171d3ca5d5ba3 P 34844c02e0f34d5fab45f43e4e5b9956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5523b2f4ad14d37ba187ff45359077e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40605 } } peers { permanent_uuid: "527d3ead14f6443b9fef7865f31d7227" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41637 } } peers { permanent_uuid: "34844c02e0f34d5fab45f43e4e5b9956" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36861 } }
I20260929 21:00:12.317471 7838 raft_consensus.cc:374] T ad9077bf2f0348f5847171d3ca5d5ba3 P 34844c02e0f34d5fab45f43e4e5b9956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:00:12.317552 7838 raft_consensus.cc:729] T ad9077bf2f0348f5847171d3ca5d5ba3 P 34844c02e0f34d5fab45f43e4e5b9956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34844c02e0f34d5fab45f43e4e5b9956, State: Initialized, Role: FOLLOWER
I20260929 21:00:12.318243 7838 consensus_queue.cc:260] T ad9077bf2f0348f5847171d3ca5d5ba3 P 34844c02e0f34d5fab45f43e4e5b9956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5523b2f4ad14d37ba187ff45359077e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40605 } } peers { permanent_uuid: "527d3ead14f6443b9fef7865f31d7227" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41637 } } peers { permanent_uuid: "34844c02e0f34d5fab45f43e4e5b9956" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36861 } }
I20260929 21:00:12.322541 7838 ts_tablet_manager.cc:1428] T ad9077bf2f0348f5847171d3ca5d5ba3 P 34844c02e0f34d5fab45f43e4e5b9956: Time spent starting tablet: real 0.016s user 0.006s sys 0.010s
I20260929 21:00:12.322815 7837 raft_consensus.cc:348] T ad9077bf2f0348f5847171d3ca5d5ba3 P 527d3ead14f6443b9fef7865f31d7227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5523b2f4ad14d37ba187ff45359077e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40605 } } peers { permanent_uuid: "527d3ead14f6443b9fef7865f31d7227" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41637 } } peers { permanent_uuid: "34844c02e0f34d5fab45f43e4e5b9956" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36861 } }
I20260929 21:00:12.323105 7803 heartbeater.cc:503] Master 127.0.97.62:38557 was elected leader, sending a full tablet report...
I20260929 21:00:12.323226 7837 raft_consensus.cc:374] T ad9077bf2f0348f5847171d3ca5d5ba3 P 527d3ead14f6443b9fef7865f31d7227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:00:12.323320 7837 raft_consensus.cc:729] T ad9077bf2f0348f5847171d3ca5d5ba3 P 527d3ead14f6443b9fef7865f31d7227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 527d3ead14f6443b9fef7865f31d7227, State: Initialized, Role: FOLLOWER
I20260929 21:00:12.323886 7837 consensus_queue.cc:260] T ad9077bf2f0348f5847171d3ca5d5ba3 P 527d3ead14f6443b9fef7865f31d7227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5523b2f4ad14d37ba187ff45359077e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40605 } } peers { permanent_uuid: "527d3ead14f6443b9fef7865f31d7227" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41637 } } peers { permanent_uuid: "34844c02e0f34d5fab45f43e4e5b9956" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36861 } }
I20260929 21:00:12.326913 7837 ts_tablet_manager.cc:1428] T ad9077bf2f0348f5847171d3ca5d5ba3 P 527d3ead14f6443b9fef7865f31d7227: Time spent starting tablet: real 0.020s user 0.009s sys 0.007s
W20260929 21:00:12.328082 7663 tablet.cc:2369] T ad9077bf2f0348f5847171d3ca5d5ba3 P c5523b2f4ad14d37ba187ff45359077e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 21:00:12.385588 7842 raft_consensus.cc:482] T ad9077bf2f0348f5847171d3ca5d5ba3 P c5523b2f4ad14d37ba187ff45359077e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 21:00:12.385917 7842 raft_consensus.cc:504] T ad9077bf2f0348f5847171d3ca5d5ba3 P c5523b2f4ad14d37ba187ff45359077e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5523b2f4ad14d37ba187ff45359077e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40605 } } peers { permanent_uuid: "527d3ead14f6443b9fef7865f31d7227" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41637 } } peers { permanent_uuid: "34844c02e0f34d5fab45f43e4e5b9956" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36861 } }
I20260929 21:00:12.387518 7842 leader_election.cc:288] T ad9077bf2f0348f5847171d3ca5d5ba3 P c5523b2f4ad14d37ba187ff45359077e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 527d3ead14f6443b9fef7865f31d7227 (127.0.97.1:41637), 34844c02e0f34d5fab45f43e4e5b9956 (127.0.97.3:36861)
I20260929 21:00:12.394752 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:12.387977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58017 (local address 127.0.97.1:41637)
0929 21:00:12.388315 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:12.388320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:12.389361 (+ 1041us) server_negotiation.cc:407] Connection header received
0929 21:00:12.389507 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:12.389512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:12.389594 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:12.389679 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:00:12.390286 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.391035 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.392985 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.393782 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.393878 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:12.393987 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:12.394359 (+ 372us) server_negotiation.cc:299] Negotiation successful
0929 21:00:12.394492 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227}
I20260929 21:00:12.394757 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:12.387993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48747 (local address 127.0.97.3:36861)
0929 21:00:12.388183 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:12.388187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:12.389496 (+ 1309us) server_negotiation.cc:407] Connection header received
0929 21:00:12.389677 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:12.389683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:12.389757 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:12.389859 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:00:12.390381 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.391062 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.393056 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.393829 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.393984 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:12.394084 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:12.394410 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 21:00:12.394528 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 21:00:12.394889 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:12.388209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41637 (local address 127.0.97.2:58017)
0929 21:00:12.389245 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:12.389261 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:12.389380 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:12.389719 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:12.389724 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:12.389749 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:00:12.390137 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:12.390147 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.391271 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.391276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:12.392800 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:12.392811 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.393981 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.393987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:12.394167 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:12.394224 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 21:00:12.394296 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 21:00:12.394448 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":919,"mutex_wait_us":59,"thread_start_us":339,"threads_started":1}
I20260929 21:00:12.395229 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:12.388209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36861 (local address 127.0.97.2:48747)
0929 21:00:12.389362 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:12.389382 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:12.389513 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:12.389885 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:12.389888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:12.389906 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:00:12.390178 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:12.390185 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.391315 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.391322 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:12.392786 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:12.392804 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.394082 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.394087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:12.394230 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:12.394252 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 21:00:12.394362 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 21:00:12.394510 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1018,"mutex_wait_us":128,"thread_start_us":671,"threads_started":1}
I20260929 21:00:12.395936 7467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad9077bf2f0348f5847171d3ca5d5ba3" candidate_uuid: "c5523b2f4ad14d37ba187ff45359077e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "527d3ead14f6443b9fef7865f31d7227" is_pre_election: true
I20260929 21:00:12.396116 7757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad9077bf2f0348f5847171d3ca5d5ba3" candidate_uuid: "c5523b2f4ad14d37ba187ff45359077e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34844c02e0f34d5fab45f43e4e5b9956" is_pre_election: true
I20260929 21:00:12.396437 7467 raft_consensus.cc:2454] T ad9077bf2f0348f5847171d3ca5d5ba3 P 527d3ead14f6443b9fef7865f31d7227 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5523b2f4ad14d37ba187ff45359077e in term 0.
I20260929 21:00:12.396621 7757 raft_consensus.cc:2454] T ad9077bf2f0348f5847171d3ca5d5ba3 P 34844c02e0f34d5fab45f43e4e5b9956 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5523b2f4ad14d37ba187ff45359077e in term 0.
I20260929 21:00:12.397167 7548 leader_election.cc:302] T ad9077bf2f0348f5847171d3ca5d5ba3 P c5523b2f4ad14d37ba187ff45359077e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 34844c02e0f34d5fab45f43e4e5b9956, 527d3ead14f6443b9fef7865f31d7227, c5523b2f4ad14d37ba187ff45359077e; no voters:
I20260929 21:00:12.397607 7842 raft_consensus.cc:2781] T ad9077bf2f0348f5847171d3ca5d5ba3 P c5523b2f4ad14d37ba187ff45359077e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 21:00:12.398141 7842 raft_consensus.cc:482] T ad9077bf2f0348f5847171d3ca5d5ba3 P c5523b2f4ad14d37ba187ff45359077e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 21:00:12.398247 7842 raft_consensus.cc:3037] T ad9077bf2f0348f5847171d3ca5d5ba3 P c5523b2f4ad14d37ba187ff45359077e [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:00:12.400617 7842 raft_consensus.cc:504] T ad9077bf2f0348f5847171d3ca5d5ba3 P c5523b2f4ad14d37ba187ff45359077e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5523b2f4ad14d37ba187ff45359077e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40605 } } peers { permanent_uuid: "527d3ead14f6443b9fef7865f31d7227" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41637 } } peers { permanent_uuid: "34844c02e0f34d5fab45f43e4e5b9956" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36861 } }
I20260929 21:00:12.401168 7842 leader_election.cc:288] T ad9077bf2f0348f5847171d3ca5d5ba3 P c5523b2f4ad14d37ba187ff45359077e [CANDIDATE]: Term 1 election: Requested vote from peers 527d3ead14f6443b9fef7865f31d7227 (127.0.97.1:41637), 34844c02e0f34d5fab45f43e4e5b9956 (127.0.97.3:36861)
I20260929 21:00:12.401638 7757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad9077bf2f0348f5847171d3ca5d5ba3" candidate_uuid: "c5523b2f4ad14d37ba187ff45359077e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34844c02e0f34d5fab45f43e4e5b9956"
I20260929 21:00:12.401638 7467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad9077bf2f0348f5847171d3ca5d5ba3" candidate_uuid: "c5523b2f4ad14d37ba187ff45359077e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "527d3ead14f6443b9fef7865f31d7227"
I20260929 21:00:12.401834 7467 raft_consensus.cc:3037] T ad9077bf2f0348f5847171d3ca5d5ba3 P 527d3ead14f6443b9fef7865f31d7227 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:00:12.401834 7757 raft_consensus.cc:3037] T ad9077bf2f0348f5847171d3ca5d5ba3 P 34844c02e0f34d5fab45f43e4e5b9956 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:00:12.404906 7757 raft_consensus.cc:2454] T ad9077bf2f0348f5847171d3ca5d5ba3 P 34844c02e0f34d5fab45f43e4e5b9956 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5523b2f4ad14d37ba187ff45359077e in term 1.
I20260929 21:00:12.404906 7467 raft_consensus.cc:2454] T ad9077bf2f0348f5847171d3ca5d5ba3 P 527d3ead14f6443b9fef7865f31d7227 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5523b2f4ad14d37ba187ff45359077e in term 1.
I20260929 21:00:12.405292 7550 leader_election.cc:302] T ad9077bf2f0348f5847171d3ca5d5ba3 P c5523b2f4ad14d37ba187ff45359077e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34844c02e0f34d5fab45f43e4e5b9956, c5523b2f4ad14d37ba187ff45359077e; no voters:
I20260929 21:00:12.405586 7842 raft_consensus.cc:2781] T ad9077bf2f0348f5847171d3ca5d5ba3 P c5523b2f4ad14d37ba187ff45359077e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:00:12.406004 7842 raft_consensus.cc:686] T ad9077bf2f0348f5847171d3ca5d5ba3 P c5523b2f4ad14d37ba187ff45359077e [term 1 LEADER]: Becoming Leader. State: Replica: c5523b2f4ad14d37ba187ff45359077e, State: Running, Role: LEADER
I20260929 21:00:12.406452 7842 consensus_queue.cc:237] T ad9077bf2f0348f5847171d3ca5d5ba3 P c5523b2f4ad14d37ba187ff45359077e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5523b2f4ad14d37ba187ff45359077e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40605 } } peers { permanent_uuid: "527d3ead14f6443b9fef7865f31d7227" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41637 } } peers { permanent_uuid: "34844c02e0f34d5fab45f43e4e5b9956" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36861 } }
W20260929 21:00:12.408412 7805 tablet.cc:2369] T ad9077bf2f0348f5847171d3ca5d5ba3 P 34844c02e0f34d5fab45f43e4e5b9956: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 21:00:12.411145 6545 catalog_manager.cc:5495] T ad9077bf2f0348f5847171d3ca5d5ba3 P c5523b2f4ad14d37ba187ff45359077e reported cstate change: term changed from 0 to 1, leader changed from <none> to c5523b2f4ad14d37ba187ff45359077e (127.0.97.2). New cstate: current_term: 1 leader_uuid: "c5523b2f4ad14d37ba187ff45359077e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5523b2f4ad14d37ba187ff45359077e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "527d3ead14f6443b9fef7865f31d7227" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34844c02e0f34d5fab45f43e4e5b9956" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36861 } health_report { overall_health: UNKNOWN } } }
I20260929 21:00:12.464666 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:12.459627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45552 (local address 127.0.97.2:40605)
0929 21:00:12.459861 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:12.459866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:12.459884 (+ 18us) server_negotiation.cc:407] Connection header received
0929 21:00:12.460099 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:12.460104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:12.460221 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:12.460367 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:00:12.460656 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.461408 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.462773 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.463160 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.463259 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:12.463633 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:00:12.464130 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:00:12.464235 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:12.464306 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 21:00:12.464365 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260929 21:00:12.471347 7515 tablet.cc:2369] T ad9077bf2f0348f5847171d3ca5d5ba3 P 527d3ead14f6443b9fef7865f31d7227: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 21:00:12.472600 7757 raft_consensus.cc:1261] T ad9077bf2f0348f5847171d3ca5d5ba3 P 34844c02e0f34d5fab45f43e4e5b9956 [term 1 FOLLOWER]: Refusing update from remote peer c5523b2f4ad14d37ba187ff45359077e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:00:12.472627 7467 raft_consensus.cc:1261] T ad9077bf2f0348f5847171d3ca5d5ba3 P 527d3ead14f6443b9fef7865f31d7227 [term 1 FOLLOWER]: Refusing update from remote peer c5523b2f4ad14d37ba187ff45359077e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:00:12.473558 7847 consensus_queue.cc:1035] T ad9077bf2f0348f5847171d3ca5d5ba3 P c5523b2f4ad14d37ba187ff45359077e [LEADER]: Connected to new peer: Peer: permanent_uuid: "34844c02e0f34d5fab45f43e4e5b9956" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:00:12.473888 7842 consensus_queue.cc:1035] T ad9077bf2f0348f5847171d3ca5d5ba3 P c5523b2f4ad14d37ba187ff45359077e [LEADER]: Connected to new peer: Peer: permanent_uuid: "527d3ead14f6443b9fef7865f31d7227" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:00:12.485448 7850 mvcc.cc:205] Tried to move back new op lower bound from 7334771148677562368 to 7334771148420026368. Current Snapshot: MvccSnapshot[applied={T|T < 7334771148677562368}]
I20260929 21:00:12.485817 7851 mvcc.cc:205] Tried to move back new op lower bound from 7334771148677562368 to 7334771148420026368. Current Snapshot: MvccSnapshot[applied={T|T < 7334771148677562368}]
I20260929 21:00:12.493727 7852 mvcc.cc:205] Tried to move back new op lower bound from 7334771148677562368 to 7334771148420026368. Current Snapshot: MvccSnapshot[applied={T|T < 7334771148677562368}]
I20260929 21:00:12.519220 6545 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 21:00:12.519486 6545 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 21:00:12.525414 6545 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b9bcd97294f4ef68830d77516fb1754: Sending DeleteTablet for 3 replicas of tablet ad9077bf2f0348f5847171d3ca5d5ba3
I20260929 21:00:12.526424 7595 tablet_service.cc:1514] Processing DeleteTablet for tablet ad9077bf2f0348f5847171d3ca5d5ba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:00:12 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45536
I20260929 21:00:12.526945 7737 tablet_service.cc:1514] Processing DeleteTablet for tablet ad9077bf2f0348f5847171d3ca5d5ba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:00:12 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43776
I20260929 21:00:12.527832 7447 tablet_service.cc:1514] Processing DeleteTablet for tablet ad9077bf2f0348f5847171d3ca5d5ba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:00:12 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46856
I20260929 21:00:12.535209 7861 tablet_replica.cc:330] T ad9077bf2f0348f5847171d3ca5d5ba3 P 34844c02e0f34d5fab45f43e4e5b9956: stopping tablet replica
I20260929 21:00:12.536381 7861 raft_consensus.cc:2229] T ad9077bf2f0348f5847171d3ca5d5ba3 P 34844c02e0f34d5fab45f43e4e5b9956 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:00:12.537640 7861 raft_consensus.cc:2258] T ad9077bf2f0348f5847171d3ca5d5ba3 P 34844c02e0f34d5fab45f43e4e5b9956 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:00:12.537719 7860 tablet_replica.cc:330] T ad9077bf2f0348f5847171d3ca5d5ba3 P c5523b2f4ad14d37ba187ff45359077e: stopping tablet replica
I20260929 21:00:12.538113 7860 raft_consensus.cc:2229] T ad9077bf2f0348f5847171d3ca5d5ba3 P c5523b2f4ad14d37ba187ff45359077e [term 1 LEADER]: Raft consensus shutting down.
I20260929 21:00:12.538781 7860 raft_consensus.cc:2258] T ad9077bf2f0348f5847171d3ca5d5ba3 P c5523b2f4ad14d37ba187ff45359077e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:00:12.540961 7860 ts_tablet_manager.cc:1905] T ad9077bf2f0348f5847171d3ca5d5ba3 P c5523b2f4ad14d37ba187ff45359077e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:00:12.541425 7862 tablet_replica.cc:330] T ad9077bf2f0348f5847171d3ca5d5ba3 P 527d3ead14f6443b9fef7865f31d7227: stopping tablet replica
I20260929 21:00:12.541854 7862 raft_consensus.cc:2229] T ad9077bf2f0348f5847171d3ca5d5ba3 P 527d3ead14f6443b9fef7865f31d7227 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:00:12.542034 7861 ts_tablet_manager.cc:1905] T ad9077bf2f0348f5847171d3ca5d5ba3 P 34844c02e0f34d5fab45f43e4e5b9956: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:00:12.542410 7862 raft_consensus.cc:2258] T ad9077bf2f0348f5847171d3ca5d5ba3 P 527d3ead14f6443b9fef7865f31d7227 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:00:12.544308 7862 ts_tablet_manager.cc:1905] T ad9077bf2f0348f5847171d3ca5d5ba3 P 527d3ead14f6443b9fef7865f31d7227: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:00:12.548203 7860 ts_tablet_manager.cc:1918] T ad9077bf2f0348f5847171d3ca5d5ba3 P c5523b2f4ad14d37ba187ff45359077e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:00:12.548368 7860 log.cc:1196] T ad9077bf2f0348f5847171d3ca5d5ba3 P c5523b2f4ad14d37ba187ff45359077e: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-1/wal/wals/ad9077bf2f0348f5847171d3ca5d5ba3
I20260929 21:00:12.548987 7860 ts_tablet_manager.cc:1939] T ad9077bf2f0348f5847171d3ca5d5ba3 P c5523b2f4ad14d37ba187ff45359077e: Deleting consensus metadata
I20260929 21:00:12.550349 7861 ts_tablet_manager.cc:1918] T ad9077bf2f0348f5847171d3ca5d5ba3 P 34844c02e0f34d5fab45f43e4e5b9956: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:00:12.550472 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 6513
I20260929 21:00:12.550508 7861 log.cc:1196] T ad9077bf2f0348f5847171d3ca5d5ba3 P 34844c02e0f34d5fab45f43e4e5b9956: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-2/wal/wals/ad9077bf2f0348f5847171d3ca5d5ba3
I20260929 21:00:12.551600 6575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:00:12.551610 7861 ts_tablet_manager.cc:1939] T ad9077bf2f0348f5847171d3ca5d5ba3 P 34844c02e0f34d5fab45f43e4e5b9956: Deleting consensus metadata
I20260929 21:00:12.553589 6531 catalog_manager.cc:4841] TS 34844c02e0f34d5fab45f43e4e5b9956 (127.0.97.3:36861): tablet ad9077bf2f0348f5847171d3ca5d5ba3 (table test-table [id=f8938ae7ce98473ab4c3917bb23f95c6]) successfully deleted
I20260929 21:00:12.558140 7862 ts_tablet_manager.cc:1918] T ad9077bf2f0348f5847171d3ca5d5ba3 P 527d3ead14f6443b9fef7865f31d7227: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:00:12.558298 7862 log.cc:1196] T ad9077bf2f0348f5847171d3ca5d5ba3 P 527d3ead14f6443b9fef7865f31d7227: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-0/wal/wals/ad9077bf2f0348f5847171d3ca5d5ba3
I20260929 21:00:12.558838 7862 ts_tablet_manager.cc:1939] T ad9077bf2f0348f5847171d3ca5d5ba3 P 527d3ead14f6443b9fef7865f31d7227: Deleting consensus metadata
I20260929 21:00:12.571516 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:12.530061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36600 (local address 127.0.97.61:44943)
0929 21:00:12.530319 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:12.530328 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:12.548589 (+ 18261us) server_negotiation.cc:407] Connection header received
0929 21:00:12.548932 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:12.548939 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:12.549019 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:12.549092 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:12.554351 (+ 5259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.555412 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.557699 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.558361 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.558483 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:12.566124 (+ 7641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:12.566129 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:12.566132 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:12.566166 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:12.568961 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:12.569600 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:12.569605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:12.569609 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:12.569729 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:12.570102 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:12.570107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:12.570110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:12.570380 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:12.570646 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:12.570963 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 21:00:12.571177 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 21:00:12.586966 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:12.549358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60472 (local address 127.0.97.60:44957)
0929 21:00:12.549548 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:12.549554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:12.571658 (+ 22104us) server_negotiation.cc:407] Connection header received
0929 21:00:12.571889 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:12.571897 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:12.572000 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:12.572078 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:12.573805 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.574482 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.575711 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.576398 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.576520 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:12.580371 (+ 3851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:12.580377 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:12.580383 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:12.580425 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:12.583817 (+ 3392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:12.584628 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:12.584634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:12.584638 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:12.584796 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:12.585297 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:12.585302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:12.585305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:12.585646 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:12.585873 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:12.586407 (+ 534us) server_negotiation.cc:299] Negotiation successful
0929 21:00:12.586619 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 21:00:12.689787 6529 catalog_manager.cc:4841] TS c5523b2f4ad14d37ba187ff45359077e (127.0.97.2:40605): tablet ad9077bf2f0348f5847171d3ca5d5ba3 (table test-table [id=f8938ae7ce98473ab4c3917bb23f95c6]) successfully deleted
W20260929 21:00:12.690037 6529 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.97.2:40605, user_credentials={real_user=slave}} blocked reactor thread for 138076us
I20260929 21:00:12.690433 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 6513
I20260929 21:00:12.690467 6529 catalog_manager.cc:4841] TS 527d3ead14f6443b9fef7865f31d7227 (127.0.97.1:41637): tablet ad9077bf2f0348f5847171d3ca5d5ba3 (table test-table [id=f8938ae7ce98473ab4c3917bb23f95c6]) successfully deleted
I20260929 21:00:12.740512 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:12.716257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36608 (local address 127.0.97.61:44943)
0929 21:00:12.716483 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:12.716493 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:12.717400 (+ 907us) server_negotiation.cc:407] Connection header received
0929 21:00:12.717672 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:12.717679 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:12.717763 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:12.717855 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:12.720754 (+ 2899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.722426 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.724938 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.725647 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.725801 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:12.734429 (+ 8628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:12.734434 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:12.734437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:12.734469 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:12.737710 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:12.738511 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:12.738517 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:12.738521 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:12.738678 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:12.739092 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:12.739099 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:12.739104 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:12.739391 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:12.739693 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:12.739901 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 21:00:12.740119 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 21:00:12.740945 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:12.717639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60478 (local address 127.0.97.60:44957)
0929 21:00:12.717803 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:12.717810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:12.725412 (+ 7602us) server_negotiation.cc:407] Connection header received
0929 21:00:12.725830 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:12.725837 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:12.725914 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:12.725983 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:12.728031 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.728718 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.730445 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:12.730804 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:12.730894 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:12.735082 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:12.735091 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:12.735099 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:12.735140 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:12.737818 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:12.738659 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:12.738664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:12.738667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:12.738798 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:12.739158 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:12.739163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:12.739167 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:12.739514 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:12.739728 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:12.740380 (+ 652us) server_negotiation.cc:299] Negotiation successful
0929 21:00:12.740587 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 21:00:13.501061 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:13.499990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38557 (local address 127.0.97.1:56555)
0929 21:00:13.500709 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:13.500877 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":477,"threads_started":1}
W20260929 21:00:13.501518 7513 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38557 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:38557: connect: Connection refused (error 111)
I20260929 21:00:13.502732 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:13.501977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38557 (local address 127.0.97.3:36459)
0929 21:00:13.502356 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:13.502511 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":197,"threads_started":1}
W20260929 21:00:13.503233 7803 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38557 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:38557: connect: Connection refused (error 111)
I20260929 21:00:13.503858 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:13.503175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38557 (local address 127.0.97.2:50509)
0929 21:00:13.503568 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:13.503704 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"threads_started":1}
W20260929 21:00:13.504315 7661 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38557 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:38557: connect: Connection refused (error 111)
I20260929 21:00:14.026261 7875 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b9bcd97294f4ef68830d77516fb1754)
I20260929 21:00:14.026609 7875 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: false } } peers { permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44957 } attrs { promote: false } }
I20260929 21:00:14.026758 7876 raft_consensus.cc:482] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b9bcd97294f4ef68830d77516fb1754)
I20260929 21:00:14.027040 7876 raft_consensus.cc:504] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: false } } peers { permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44957 } attrs { promote: false } }
I20260929 21:00:14.028038 7875 leader_election.cc:288] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b9bcd97294f4ef68830d77516fb1754 (127.0.97.62:38557), b72fa9367117481cb47b95f56114e583 (127.0.97.61:44943)
I20260929 21:00:14.028204 7876 leader_election.cc:288] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b9bcd97294f4ef68830d77516fb1754 (127.0.97.62:38557), 46eee36a2fe34cd8b2582f89c8f11aff (127.0.97.60:44957)
I20260929 21:00:14.030560 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.028409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38557 (local address 127.0.0.1:60800)
0929 21:00:14.030116 (+ 1707us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:14.030308 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1608,"thread_start_us":1501,"threads_started":1}
W20260929 21:00:14.031473 7128 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38557: connect: Connection refused (error 111)
I20260929 21:00:14.032657 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.028115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38557 (local address 127.0.0.1:60794)
0929 21:00:14.032239 (+ 4124us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:14.032412 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2978,"thread_start_us":348,"threads_started":1}
W20260929 21:00:14.032888 7319 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38557: connect: Connection refused (error 111)
I20260929 21:00:14.033938 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.033077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38557 (local address 127.0.0.1:60804)
0929 21:00:14.033503 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:14.033684 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278}
W20260929 21:00:14.034190 7128 leader_election.cc:334] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4b9bcd97294f4ef68830d77516fb1754 (127.0.97.62:38557): Network error: Client connection negotiation failed: client connection to 127.0.97.62:38557: connect: Connection refused (error 111)
I20260929 21:00:14.035097 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.034695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38557 (local address 127.0.0.1:60806)
0929 21:00:14.034862 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:14.034947 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 21:00:14.035277 7319 leader_election.cc:334] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4b9bcd97294f4ef68830d77516fb1754 (127.0.97.62:38557): Network error: Client connection negotiation failed: client connection to 127.0.97.62:38557: connect: Connection refused (error 111)
I20260929 21:00:14.037760 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.028362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44943 (local address 127.0.0.1:36610)
0929 21:00:14.029749 (+ 1387us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:14.029767 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:14.029968 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:14.030515 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:14.030521 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:14.030554 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:00:14.031520 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:14.031540 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:14.033342 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:14.033348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:14.035756 (+ 2408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:14.035767 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:14.036950 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:14.036956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:14.037119 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:14.037142 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 21:00:14.037265 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 21:00:14.037414 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1269,"thread_start_us":3637,"threads_started":1}
I20260929 21:00:14.037786 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.028307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36610 (local address 127.0.97.61:44943)
0929 21:00:14.029326 (+ 1019us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:14.029335 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:14.029856 (+ 521us) server_negotiation.cc:407] Connection header received
0929 21:00:14.030125 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:14.030131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:14.030201 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:14.030287 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:00:14.031795 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:14.033086 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:14.035901 (+ 2815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:14.036726 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:14.036914 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:14.037007 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:14.037290 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 21:00:14.037417 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":888,"thread_start_us":731,"threads_started":1}
I20260929 21:00:14.038522 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.030326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44957 (local address 127.0.0.1:60490)
0929 21:00:14.030749 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:14.030764 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:14.030869 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:14.032661 (+ 1792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:14.032668 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:14.032703 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:00:14.033005 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:14.033017 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:14.035088 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:14.035096 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:14.036454 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:14.036465 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:14.037694 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:14.037700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:14.037937 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:14.037974 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 21:00:14.038085 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 21:00:14.038216 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":1,"thread_start_us":813,"threads_started":1}
I20260929 21:00:14.038401 7152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b72fa9367117481cb47b95f56114e583" is_pre_election: true
I20260929 21:00:14.038802 7152 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46eee36a2fe34cd8b2582f89c8f11aff in term 1.
I20260929 21:00:14.038908 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.030861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60490 (local address 127.0.97.60:44957)
0929 21:00:14.031707 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:14.031712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:14.031728 (+ 16us) server_negotiation.cc:407] Connection header received
0929 21:00:14.031782 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:14.031787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:14.031854 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:14.033150 (+ 1296us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:00:14.033369 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:14.034959 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:14.036606 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:14.037497 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:14.037648 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:14.037761 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:14.038102 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 21:00:14.038633 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":389,"server-negotiator.queue_time_us":703,"thread_start_us":1075,"threads_started":1}
I20260929 21:00:14.039381 7318 leader_election.cc:302] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46eee36a2fe34cd8b2582f89c8f11aff, b72fa9367117481cb47b95f56114e583; no voters: 4b9bcd97294f4ef68830d77516fb1754
I20260929 21:00:14.039409 7343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b72fa9367117481cb47b95f56114e583" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" is_pre_election: true
I20260929 21:00:14.039711 7875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 21:00:14.039845 7875 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4b9bcd97294f4ef68830d77516fb1754)
I20260929 21:00:14.039966 7875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [term 1 FOLLOWER]: Advancing to term 2
I20260929 21:00:14.042255 7875 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: false } } peers { permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44957 } attrs { promote: false } }
I20260929 21:00:14.042657 7343 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b72fa9367117481cb47b95f56114e583 in current term 2: Already voted for candidate 46eee36a2fe34cd8b2582f89c8f11aff in this term.
I20260929 21:00:14.042917 7875 leader_election.cc:288] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [CANDIDATE]: Term 2 election: Requested vote from peers 4b9bcd97294f4ef68830d77516fb1754 (127.0.97.62:38557), b72fa9367117481cb47b95f56114e583 (127.0.97.61:44943)
I20260929 21:00:14.043323 7152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b72fa9367117481cb47b95f56114e583"
I20260929 21:00:14.043499 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.042985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38557 (local address 127.0.0.1:60808)
0929 21:00:14.043231 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:14.043364 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260929 21:00:14.043566 7152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [term 1 FOLLOWER]: Advancing to term 2
I20260929 21:00:14.043645 7128 leader_election.cc:302] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b72fa9367117481cb47b95f56114e583; no voters: 46eee36a2fe34cd8b2582f89c8f11aff, 4b9bcd97294f4ef68830d77516fb1754
I20260929 21:00:14.044488 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.044088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38557 (local address 127.0.0.1:60812)
0929 21:00:14.044228 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:14.044328 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 21:00:14.044692 7319 leader_election.cc:334] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4b9bcd97294f4ef68830d77516fb1754 (127.0.97.62:38557): Network error: Client connection negotiation failed: client connection to 127.0.97.62:38557: connect: Connection refused (error 111)
I20260929 21:00:14.046145 7152 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46eee36a2fe34cd8b2582f89c8f11aff in term 2.
I20260929 21:00:14.046361 7876 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 21:00:14.046518 7318 leader_election.cc:302] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46eee36a2fe34cd8b2582f89c8f11aff, b72fa9367117481cb47b95f56114e583; no voters: 4b9bcd97294f4ef68830d77516fb1754
I20260929 21:00:14.046816 7875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [term 2 FOLLOWER]: Leader election won for term 2
I20260929 21:00:14.047263 7875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [term 2 LEADER]: Becoming Leader. State: Replica: 46eee36a2fe34cd8b2582f89c8f11aff, State: Running, Role: LEADER
I20260929 21:00:14.047623 7875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: false } } peers { permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44957 } attrs { promote: false } }
I20260929 21:00:14.049431 7885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46eee36a2fe34cd8b2582f89c8f11aff. Latest consensus state: current_term: 2 leader_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: false } } peers { permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44957 } attrs { promote: false } } }
I20260929 21:00:14.049808 7885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [sys.catalog]: This master's current role is: LEADER
I20260929 21:00:14.050499 7887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:00:14.054051 7887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:00:14.054780 7887 catalog_manager.cc:1259] Loaded cluster ID: 1d78b66f93774de588144898a42be9a5
I20260929 21:00:14.054849 7887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:00:14.055862 7887 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:00:14.056578 7887 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff: Loaded TSK: 0
I20260929 21:00:14.057442 7887 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 21:00:14.074189 7332 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60478:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 21:00:14.081219 7332 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 21:00:14.085386 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.084968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38557 (local address 127.0.0.1:60828)
0929 21:00:14.085139 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:14.085213 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 21:00:14.085417 7152 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [term 2 FOLLOWER]: Refusing update from remote peer 46eee36a2fe34cd8b2582f89c8f11aff: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260929 21:00:14.086294 7885 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [LEADER]: Connected to new peer: Peer: permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260929 21:00:14.086587 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.086069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38557 (local address 127.0.0.1:60830)
0929 21:00:14.086336 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:14.086434 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260929 21:00:14.086752 7319 consensus_peers.cc:487] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff -> Peer 4b9bcd97294f4ef68830d77516fb1754 (127.0.97.62:38557): Couldn't send request to peer 4b9bcd97294f4ef68830d77516fb1754. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38557: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 21:00:14.089222 7876 sys_catalog.cc:455] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46eee36a2fe34cd8b2582f89c8f11aff. Latest consensus state: current_term: 2 leader_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: false } } peers { permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44957 } attrs { promote: false } } }
I20260929 21:00:14.089536 7876 sys_catalog.cc:458] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 21:00:14.090938 7875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: false } } peers { permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44957 } attrs { promote: false } } }
I20260929 21:00:14.091343 7875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff [sys.catalog]: This master's current role is: LEADER
I20260929 21:00:14.091741 7876 sys_catalog.cc:455] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b9bcd97294f4ef68830d77516fb1754" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38557 } } peers { permanent_uuid: "b72fa9367117481cb47b95f56114e583" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44943 } attrs { promote: false } } peers { permanent_uuid: "46eee36a2fe34cd8b2582f89c8f11aff" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44957 } attrs { promote: false } } }
I20260929 21:00:14.092011 7876 sys_catalog.cc:458] T 00000000000000000000000000000000 P b72fa9367117481cb47b95f56114e583 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 21:00:14.107568 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.099852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43792 (local address 127.0.97.3:36861)
0929 21:00:14.102276 (+ 2424us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:14.102280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:14.102294 (+ 14us) server_negotiation.cc:407] Connection header received
0929 21:00:14.102350 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:14.102353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:14.102446 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:14.103006 (+ 560us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:00:14.103139 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:14.103820 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:14.105616 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:14.106622 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:14.106788 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:14.106897 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:14.107176 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 21:00:14.107308 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2310,"thread_start_us":2224,"threads_started":1}
I20260929 21:00:14.107573 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.099742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36861 (local address 127.0.0.1:43792)
0929 21:00:14.099889 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:14.099906 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:14.100013 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:14.102588 (+ 2575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:14.102593 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:14.102619 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:00:14.102895 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:14.102904 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:14.103947 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:14.103955 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:14.105432 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:14.105446 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:14.106836 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:14.106842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:14.107015 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:14.107039 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 21:00:14.107140 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 21:00:14.107286 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 21:00:14.108445 7737 tablet_service.cc:1467] Processing CreateTablet for tablet 971c2f439d264794bc0ce7550b6690cc (DEFAULT_TABLE table=test-table [id=776e150f6bad4c2baaff928ee29260b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:00:14.108944 7737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 971c2f439d264794bc0ce7550b6690cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:00:14.115101 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.099218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41637 (local address 127.0.0.1:46870)
0929 21:00:14.099360 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:14.099381 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:14.099486 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:14.109718 (+ 10232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:14.109722 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:14.109739 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:00:14.109958 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:14.109970 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:14.111528 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:14.111536 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:14.113110 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:14.113121 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:14.114348 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:14.114353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:14.114534 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:14.114580 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 21:00:14.114678 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 21:00:14.114815 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260929 21:00:14.115124 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.099293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46870 (local address 127.0.97.1:41637)
0929 21:00:14.109423 (+ 10130us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:14.109429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:14.109446 (+ 17us) server_negotiation.cc:407] Connection header received
0929 21:00:14.109513 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:14.109519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:14.109588 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:14.110110 (+ 522us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:00:14.110277 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:14.111370 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:14.113240 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:14.114189 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:14.114346 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:14.114448 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:14.114710 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 21:00:14.114840 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10027,"thread_start_us":10227,"threads_started":1}
I20260929 21:00:14.115873 7447 tablet_service.cc:1467] Processing CreateTablet for tablet 971c2f439d264794bc0ce7550b6690cc (DEFAULT_TABLE table=test-table [id=776e150f6bad4c2baaff928ee29260b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:00:14.116341 7447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 971c2f439d264794bc0ce7550b6690cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:00:14.117535 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.100624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40605 (local address 127.0.0.1:45554)
0929 21:00:14.101332 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:14.101353 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:14.101479 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:14.109685 (+ 8206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:14.109693 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:14.109719 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:00:14.110099 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:14.110109 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:14.111603 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:14.111609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:14.113870 (+ 2261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:14.113884 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:14.116711 (+ 2827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:14.116717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:14.116949 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:14.116981 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 21:00:14.117079 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 21:00:14.117210 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":652,"threads_started":1}
I20260929 21:00:14.117775 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.101158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45554 (local address 127.0.97.2:40605)
0929 21:00:14.109236 (+ 8078us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:14.109244 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:14.109267 (+ 23us) server_negotiation.cc:407] Connection header received
0929 21:00:14.109345 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:14.109351 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:14.109438 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:14.110262 (+ 824us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:00:14.110433 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:14.111462 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:14.114049 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:14.116498 (+ 2449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:14.116669 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:14.116819 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:14.117136 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 21:00:14.117351 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7939,"thread_start_us":7835,"threads_started":1}
I20260929 21:00:14.118763 7595 tablet_service.cc:1467] Processing CreateTablet for tablet 971c2f439d264794bc0ce7550b6690cc (DEFAULT_TABLE table=test-table [id=776e150f6bad4c2baaff928ee29260b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:00:14.119321 7595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 971c2f439d264794bc0ce7550b6690cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:00:14.122326 7898 tablet_bootstrap.cc:492] T 971c2f439d264794bc0ce7550b6690cc P 34844c02e0f34d5fab45f43e4e5b9956: Bootstrap starting.
I20260929 21:00:14.124104 7898 tablet_bootstrap.cc:654] T 971c2f439d264794bc0ce7550b6690cc P 34844c02e0f34d5fab45f43e4e5b9956: Neither blocks nor log segments found. Creating new log.
I20260929 21:00:14.125409 7899 tablet_bootstrap.cc:492] T 971c2f439d264794bc0ce7550b6690cc P 527d3ead14f6443b9fef7865f31d7227: Bootstrap starting.
I20260929 21:00:14.126504 7901 tablet_bootstrap.cc:492] T 971c2f439d264794bc0ce7550b6690cc P c5523b2f4ad14d37ba187ff45359077e: Bootstrap starting.
I20260929 21:00:14.127275 7899 tablet_bootstrap.cc:654] T 971c2f439d264794bc0ce7550b6690cc P 527d3ead14f6443b9fef7865f31d7227: Neither blocks nor log segments found. Creating new log.
I20260929 21:00:14.128310 7901 tablet_bootstrap.cc:654] T 971c2f439d264794bc0ce7550b6690cc P c5523b2f4ad14d37ba187ff45359077e: Neither blocks nor log segments found. Creating new log.
I20260929 21:00:14.129565 7899 tablet_bootstrap.cc:492] T 971c2f439d264794bc0ce7550b6690cc P 527d3ead14f6443b9fef7865f31d7227: No bootstrap required, opened a new log
I20260929 21:00:14.129699 7899 ts_tablet_manager.cc:1397] T 971c2f439d264794bc0ce7550b6690cc P 527d3ead14f6443b9fef7865f31d7227: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 21:00:14.130424 7901 tablet_bootstrap.cc:492] T 971c2f439d264794bc0ce7550b6690cc P c5523b2f4ad14d37ba187ff45359077e: No bootstrap required, opened a new log
I20260929 21:00:14.130571 7901 ts_tablet_manager.cc:1397] T 971c2f439d264794bc0ce7550b6690cc P c5523b2f4ad14d37ba187ff45359077e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260929 21:00:14.130540 7899 raft_consensus.cc:348] T 971c2f439d264794bc0ce7550b6690cc P 527d3ead14f6443b9fef7865f31d7227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527d3ead14f6443b9fef7865f31d7227" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41637 } } peers { permanent_uuid: "34844c02e0f34d5fab45f43e4e5b9956" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36861 } } peers { permanent_uuid: "c5523b2f4ad14d37ba187ff45359077e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40605 } }
I20260929 21:00:14.130821 7899 raft_consensus.cc:374] T 971c2f439d264794bc0ce7550b6690cc P 527d3ead14f6443b9fef7865f31d7227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:00:14.130918 7899 raft_consensus.cc:729] T 971c2f439d264794bc0ce7550b6690cc P 527d3ead14f6443b9fef7865f31d7227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 527d3ead14f6443b9fef7865f31d7227, State: Initialized, Role: FOLLOWER
I20260929 21:00:14.131152 7899 consensus_queue.cc:260] T 971c2f439d264794bc0ce7550b6690cc P 527d3ead14f6443b9fef7865f31d7227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527d3ead14f6443b9fef7865f31d7227" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41637 } } peers { permanent_uuid: "34844c02e0f34d5fab45f43e4e5b9956" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36861 } } peers { permanent_uuid: "c5523b2f4ad14d37ba187ff45359077e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40605 } }
I20260929 21:00:14.131340 7901 raft_consensus.cc:348] T 971c2f439d264794bc0ce7550b6690cc P c5523b2f4ad14d37ba187ff45359077e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527d3ead14f6443b9fef7865f31d7227" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41637 } } peers { permanent_uuid: "34844c02e0f34d5fab45f43e4e5b9956" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36861 } } peers { permanent_uuid: "c5523b2f4ad14d37ba187ff45359077e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40605 } }
I20260929 21:00:14.131592 7901 raft_consensus.cc:374] T 971c2f439d264794bc0ce7550b6690cc P c5523b2f4ad14d37ba187ff45359077e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:00:14.131688 7901 raft_consensus.cc:729] T 971c2f439d264794bc0ce7550b6690cc P c5523b2f4ad14d37ba187ff45359077e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5523b2f4ad14d37ba187ff45359077e, State: Initialized, Role: FOLLOWER
I20260929 21:00:14.131979 7899 ts_tablet_manager.cc:1428] T 971c2f439d264794bc0ce7550b6690cc P 527d3ead14f6443b9fef7865f31d7227: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 21:00:14.131897 7901 consensus_queue.cc:260] T 971c2f439d264794bc0ce7550b6690cc P c5523b2f4ad14d37ba187ff45359077e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527d3ead14f6443b9fef7865f31d7227" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41637 } } peers { permanent_uuid: "34844c02e0f34d5fab45f43e4e5b9956" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36861 } } peers { permanent_uuid: "c5523b2f4ad14d37ba187ff45359077e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40605 } }
I20260929 21:00:14.133476 7898 tablet_bootstrap.cc:492] T 971c2f439d264794bc0ce7550b6690cc P 34844c02e0f34d5fab45f43e4e5b9956: No bootstrap required, opened a new log
I20260929 21:00:14.133668 7898 ts_tablet_manager.cc:1397] T 971c2f439d264794bc0ce7550b6690cc P 34844c02e0f34d5fab45f43e4e5b9956: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260929 21:00:14.134905 7898 raft_consensus.cc:348] T 971c2f439d264794bc0ce7550b6690cc P 34844c02e0f34d5fab45f43e4e5b9956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527d3ead14f6443b9fef7865f31d7227" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41637 } } peers { permanent_uuid: "34844c02e0f34d5fab45f43e4e5b9956" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36861 } } peers { permanent_uuid: "c5523b2f4ad14d37ba187ff45359077e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40605 } }
I20260929 21:00:14.135279 7898 raft_consensus.cc:374] T 971c2f439d264794bc0ce7550b6690cc P 34844c02e0f34d5fab45f43e4e5b9956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:00:14.136008 7512 heartbeater.cc:503] Master 127.0.97.60:44957 was elected leader, sending a full tablet report...
I20260929 21:00:14.136574 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.132624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38557 (local address 127.0.97.1:33251)
0929 21:00:14.136005 (+ 3381us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:14.136193 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3191,"thread_start_us":2998,"threads_started":1}
I20260929 21:00:14.136981 7898 raft_consensus.cc:729] T 971c2f439d264794bc0ce7550b6690cc P 34844c02e0f34d5fab45f43e4e5b9956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34844c02e0f34d5fab45f43e4e5b9956, State: Initialized, Role: FOLLOWER
I20260929 21:00:14.138473 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.137904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38557 (local address 127.0.97.1:50745)
0929 21:00:14.138260 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:14.138340 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261}
I20260929 21:00:14.137320 7901 ts_tablet_manager.cc:1428] T 971c2f439d264794bc0ce7550b6690cc P c5523b2f4ad14d37ba187ff45359077e: Time spent starting tablet: real 0.007s user 0.000s sys 0.007s
I20260929 21:00:14.139246 7898 consensus_queue.cc:260] T 971c2f439d264794bc0ce7550b6690cc P 34844c02e0f34d5fab45f43e4e5b9956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527d3ead14f6443b9fef7865f31d7227" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41637 } } peers { permanent_uuid: "34844c02e0f34d5fab45f43e4e5b9956" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36861 } } peers { permanent_uuid: "c5523b2f4ad14d37ba187ff45359077e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40605 } }
I20260929 21:00:14.142741 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.137981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38557 (local address 127.0.97.2:36717)
0929 21:00:14.142321 (+ 4340us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:14.142440 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4243,"thread_start_us":4141,"threads_started":1}
I20260929 21:00:14.143254 7898 ts_tablet_manager.cc:1428] T 971c2f439d264794bc0ce7550b6690cc P 34844c02e0f34d5fab45f43e4e5b9956: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260929 21:00:14.143795 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.143323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38557 (local address 127.0.97.2:34527)
0929 21:00:14.143565 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:14.143653 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260929 21:00:14.144207 7660 heartbeater.cc:503] Master 127.0.97.60:44957 was elected leader, sending a full tablet report...
I20260929 21:00:14.145711 7802 heartbeater.cc:503] Master 127.0.97.60:44957 was elected leader, sending a full tablet report...
I20260929 21:00:14.146013 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.143784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38557 (local address 127.0.97.3:52661)
0929 21:00:14.145571 (+ 1787us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:14.145732 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1646,"thread_start_us":1512,"threads_started":1}
I20260929 21:00:14.371671 7904 raft_consensus.cc:482] T 971c2f439d264794bc0ce7550b6690cc P 527d3ead14f6443b9fef7865f31d7227 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 21:00:14.371937 7904 raft_consensus.cc:504] T 971c2f439d264794bc0ce7550b6690cc P 527d3ead14f6443b9fef7865f31d7227 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527d3ead14f6443b9fef7865f31d7227" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41637 } } peers { permanent_uuid: "34844c02e0f34d5fab45f43e4e5b9956" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36861 } } peers { permanent_uuid: "c5523b2f4ad14d37ba187ff45359077e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40605 } }
I20260929 21:00:14.373582 7904 leader_election.cc:288] T 971c2f439d264794bc0ce7550b6690cc P 527d3ead14f6443b9fef7865f31d7227 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34844c02e0f34d5fab45f43e4e5b9956 (127.0.97.3:36861), c5523b2f4ad14d37ba187ff45359077e (127.0.97.2:40605)
I20260929 21:00:14.374246 7905 raft_consensus.cc:482] T 971c2f439d264794bc0ce7550b6690cc P c5523b2f4ad14d37ba187ff45359077e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 21:00:14.374500 7905 raft_consensus.cc:504] T 971c2f439d264794bc0ce7550b6690cc P c5523b2f4ad14d37ba187ff45359077e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527d3ead14f6443b9fef7865f31d7227" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41637 } } peers { permanent_uuid: "34844c02e0f34d5fab45f43e4e5b9956" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36861 } } peers { permanent_uuid: "c5523b2f4ad14d37ba187ff45359077e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40605 } }
I20260929 21:00:14.375417 7905 leader_election.cc:288] T 971c2f439d264794bc0ce7550b6690cc P c5523b2f4ad14d37ba187ff45359077e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 527d3ead14f6443b9fef7865f31d7227 (127.0.97.1:41637), 34844c02e0f34d5fab45f43e4e5b9956 (127.0.97.3:36861)
I20260929 21:00:14.375835 7466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971c2f439d264794bc0ce7550b6690cc" candidate_uuid: "c5523b2f4ad14d37ba187ff45359077e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "527d3ead14f6443b9fef7865f31d7227" is_pre_election: true
I20260929 21:00:14.376121 7466 raft_consensus.cc:2454] T 971c2f439d264794bc0ce7550b6690cc P 527d3ead14f6443b9fef7865f31d7227 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5523b2f4ad14d37ba187ff45359077e in term 0.
I20260929 21:00:14.376147 7757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971c2f439d264794bc0ce7550b6690cc" candidate_uuid: "c5523b2f4ad14d37ba187ff45359077e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34844c02e0f34d5fab45f43e4e5b9956" is_pre_election: true
I20260929 21:00:14.376449 7757 raft_consensus.cc:2454] T 971c2f439d264794bc0ce7550b6690cc P 34844c02e0f34d5fab45f43e4e5b9956 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5523b2f4ad14d37ba187ff45359077e in term 0.
I20260929 21:00:14.376554 7548 leader_election.cc:302] T 971c2f439d264794bc0ce7550b6690cc P c5523b2f4ad14d37ba187ff45359077e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 527d3ead14f6443b9fef7865f31d7227, c5523b2f4ad14d37ba187ff45359077e; no voters:
I20260929 21:00:14.376883 7905 raft_consensus.cc:2781] T 971c2f439d264794bc0ce7550b6690cc P c5523b2f4ad14d37ba187ff45359077e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 21:00:14.376999 7905 raft_consensus.cc:482] T 971c2f439d264794bc0ce7550b6690cc P c5523b2f4ad14d37ba187ff45359077e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 21:00:14.377096 7905 raft_consensus.cc:3037] T 971c2f439d264794bc0ce7550b6690cc P c5523b2f4ad14d37ba187ff45359077e [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:00:14.379868 7905 raft_consensus.cc:504] T 971c2f439d264794bc0ce7550b6690cc P c5523b2f4ad14d37ba187ff45359077e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527d3ead14f6443b9fef7865f31d7227" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41637 } } peers { permanent_uuid: "34844c02e0f34d5fab45f43e4e5b9956" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36861 } } peers { permanent_uuid: "c5523b2f4ad14d37ba187ff45359077e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40605 } }
I20260929 21:00:14.380968 7466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971c2f439d264794bc0ce7550b6690cc" candidate_uuid: "c5523b2f4ad14d37ba187ff45359077e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "527d3ead14f6443b9fef7865f31d7227"
I20260929 21:00:14.381196 7466 raft_consensus.cc:3037] T 971c2f439d264794bc0ce7550b6690cc P 527d3ead14f6443b9fef7865f31d7227 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:00:14.381340 7757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971c2f439d264794bc0ce7550b6690cc" candidate_uuid: "c5523b2f4ad14d37ba187ff45359077e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34844c02e0f34d5fab45f43e4e5b9956"
I20260929 21:00:14.381546 7757 raft_consensus.cc:3037] T 971c2f439d264794bc0ce7550b6690cc P 34844c02e0f34d5fab45f43e4e5b9956 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:00:14.381861 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.373935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40769 (local address 127.0.97.3:36861)
0929 21:00:14.374139 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:14.374146 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:14.374173 (+ 27us) server_negotiation.cc:407] Connection header received
0929 21:00:14.374377 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:14.374406 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:14.374481 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:14.374576 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:00:14.375236 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:14.376316 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:14.379026 (+ 2710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:14.380329 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:14.380488 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:14.380589 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:14.381234 (+ 645us) server_negotiation.cc:299] Negotiation successful
0929 21:00:14.381447 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 21:00:14.382046 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.373953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43441 (local address 127.0.97.2:40605)
0929 21:00:14.374257 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:14.374265 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:14.375044 (+ 779us) server_negotiation.cc:407] Connection header received
0929 21:00:14.375182 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:14.375187 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:14.375291 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:14.375402 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:00:14.376310 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:14.377433 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:14.379592 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:14.380866 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:14.381008 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:14.381104 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:14.381676 (+ 572us) server_negotiation.cc:299] Negotiation successful
0929 21:00:14.381806 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260929 21:00:14.382664 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.373851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36861 (local address 127.0.97.1:40769)
0929 21:00:14.374036 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:14.374056 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:14.374192 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:14.374759 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:14.374764 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:14.374803 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:00:14.375118 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:14.375132 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:14.376647 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:14.376654 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:14.378871 (+ 2217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:14.378884 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:14.380512 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:14.380544 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:14.380778 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:14.380813 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 21:00:14.380934 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 21:00:14.381094 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":6}
I20260929 21:00:14.382901 7905 leader_election.cc:288] T 971c2f439d264794bc0ce7550b6690cc P c5523b2f4ad14d37ba187ff45359077e [CANDIDATE]: Term 1 election: Requested vote from peers 527d3ead14f6443b9fef7865f31d7227 (127.0.97.1:41637), 34844c02e0f34d5fab45f43e4e5b9956 (127.0.97.3:36861)
I20260929 21:00:14.382745 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.373867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40605 (local address 127.0.97.1:43441)
0929 21:00:14.374925 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:14.374946 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:14.375061 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:14.375579 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:14.375584 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:14.375612 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:00:14.376098 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:14.376120 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:14.377578 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:14.377585 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:14.379428 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:14.379444 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:14.381260 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:14.381266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:14.381479 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:14.381511 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 21:00:14.382194 (+ 683us) client_negotiation.cc:240] Negotiation successful
0929 21:00:14.382338 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"mutex_wait_us":387,"thread_start_us":375,"threads_started":1}
I20260929 21:00:14.383821 7615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971c2f439d264794bc0ce7550b6690cc" candidate_uuid: "527d3ead14f6443b9fef7865f31d7227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5523b2f4ad14d37ba187ff45359077e" is_pre_election: true
I20260929 21:00:14.383868 7756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971c2f439d264794bc0ce7550b6690cc" candidate_uuid: "527d3ead14f6443b9fef7865f31d7227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34844c02e0f34d5fab45f43e4e5b9956" is_pre_election: true
I20260929 21:00:14.384277 7615 raft_consensus.cc:2379] T 971c2f439d264794bc0ce7550b6690cc P c5523b2f4ad14d37ba187ff45359077e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 527d3ead14f6443b9fef7865f31d7227 in current term 1: Already voted for candidate c5523b2f4ad14d37ba187ff45359077e in this term.
I20260929 21:00:14.384543 7757 raft_consensus.cc:2454] T 971c2f439d264794bc0ce7550b6690cc P 34844c02e0f34d5fab45f43e4e5b9956 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5523b2f4ad14d37ba187ff45359077e in term 1.
I20260929 21:00:14.384716 7466 raft_consensus.cc:2454] T 971c2f439d264794bc0ce7550b6690cc P 527d3ead14f6443b9fef7865f31d7227 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5523b2f4ad14d37ba187ff45359077e in term 1.
I20260929 21:00:14.384943 7550 leader_election.cc:302] T 971c2f439d264794bc0ce7550b6690cc P c5523b2f4ad14d37ba187ff45359077e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34844c02e0f34d5fab45f43e4e5b9956, c5523b2f4ad14d37ba187ff45359077e; no voters:
I20260929 21:00:14.385080 7400 leader_election.cc:302] T 971c2f439d264794bc0ce7550b6690cc P 527d3ead14f6443b9fef7865f31d7227 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 527d3ead14f6443b9fef7865f31d7227; no voters: 34844c02e0f34d5fab45f43e4e5b9956, c5523b2f4ad14d37ba187ff45359077e
I20260929 21:00:14.385304 7905 raft_consensus.cc:2781] T 971c2f439d264794bc0ce7550b6690cc P c5523b2f4ad14d37ba187ff45359077e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:00:14.385470 7904 raft_consensus.cc:2726] T 971c2f439d264794bc0ce7550b6690cc P 527d3ead14f6443b9fef7865f31d7227 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 21:00:14.385803 7905 raft_consensus.cc:686] T 971c2f439d264794bc0ce7550b6690cc P c5523b2f4ad14d37ba187ff45359077e [term 1 LEADER]: Becoming Leader. State: Replica: c5523b2f4ad14d37ba187ff45359077e, State: Running, Role: LEADER
I20260929 21:00:14.386130 7905 consensus_queue.cc:237] T 971c2f439d264794bc0ce7550b6690cc P c5523b2f4ad14d37ba187ff45359077e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527d3ead14f6443b9fef7865f31d7227" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41637 } } peers { permanent_uuid: "34844c02e0f34d5fab45f43e4e5b9956" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36861 } } peers { permanent_uuid: "c5523b2f4ad14d37ba187ff45359077e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40605 } }
I20260929 21:00:14.386878 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.386426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38557 (local address 127.0.97.2:59519)
0929 21:00:14.386582 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:14.386695 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 21:00:14.388931 7332 catalog_manager.cc:5495] T 971c2f439d264794bc0ce7550b6690cc P c5523b2f4ad14d37ba187ff45359077e reported cstate change: term changed from 0 to 1, leader changed from <none> to c5523b2f4ad14d37ba187ff45359077e (127.0.97.2). New cstate: current_term: 1 leader_uuid: "c5523b2f4ad14d37ba187ff45359077e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527d3ead14f6443b9fef7865f31d7227" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34844c02e0f34d5fab45f43e4e5b9956" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5523b2f4ad14d37ba187ff45359077e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40605 } health_report { overall_health: HEALTHY } } }
I20260929 21:00:14.472622 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.468297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45570 (local address 127.0.97.2:40605)
0929 21:00:14.468447 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:14.468451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:14.468468 (+ 17us) server_negotiation.cc:407] Connection header received
0929 21:00:14.468620 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:14.468624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:14.468686 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:14.468762 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:00:14.469160 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:14.469898 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:14.471191 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:14.471567 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:14.471656 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:14.472012 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:00:14.472103 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:00:14.472169 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:14.472354 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 21:00:14.472412 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 21:00:14.475863 7757 raft_consensus.cc:1261] T 971c2f439d264794bc0ce7550b6690cc P 34844c02e0f34d5fab45f43e4e5b9956 [term 1 FOLLOWER]: Refusing update from remote peer c5523b2f4ad14d37ba187ff45359077e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:00:14.475898 7466 raft_consensus.cc:1261] T 971c2f439d264794bc0ce7550b6690cc P 527d3ead14f6443b9fef7865f31d7227 [term 1 FOLLOWER]: Refusing update from remote peer c5523b2f4ad14d37ba187ff45359077e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:00:14.476533 7911 consensus_queue.cc:1035] T 971c2f439d264794bc0ce7550b6690cc P c5523b2f4ad14d37ba187ff45359077e [LEADER]: Connected to new peer: Peer: permanent_uuid: "527d3ead14f6443b9fef7865f31d7227" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:00:14.476883 7905 consensus_queue.cc:1035] T 971c2f439d264794bc0ce7550b6690cc P c5523b2f4ad14d37ba187ff45359077e [LEADER]: Connected to new peer: Peer: permanent_uuid: "34844c02e0f34d5fab45f43e4e5b9956" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:00:14.479180 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.478572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38557 (local address 127.0.97.1:43745)
0929 21:00:14.478855 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:14.478964 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
I20260929 21:00:14.480109 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.479576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38557 (local address 127.0.97.3:39325)
0929 21:00:14.479846 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:14.479937 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
I20260929 21:00:14.487882 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.487098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38557 (local address 127.0.97.1:37241)
0929 21:00:14.487515 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:14.487628 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240}
I20260929 21:00:14.489001 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.488320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38557 (local address 127.0.97.3:35517)
0929 21:00:14.488602 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:14.488756 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260929 21:00:14.490240 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.489567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38557 (local address 127.0.97.2:37711)
0929 21:00:14.489913 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:14.490031 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188}
I20260929 21:00:14.497761 7333 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 21:00:14.498134 7333 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 21:00:14.507266 7333 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 46eee36a2fe34cd8b2582f89c8f11aff: Sending DeleteTablet for 3 replicas of tablet 971c2f439d264794bc0ce7550b6690cc
I20260929 21:00:14.508288 7447 tablet_service.cc:1514] Processing DeleteTablet for tablet 971c2f439d264794bc0ce7550b6690cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:00:14 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:46870
I20260929 21:00:14.508735 7595 tablet_service.cc:1514] Processing DeleteTablet for tablet 971c2f439d264794bc0ce7550b6690cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:00:14 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:45554
I20260929 21:00:14.509300 7923 tablet_replica.cc:330] T 971c2f439d264794bc0ce7550b6690cc P 527d3ead14f6443b9fef7865f31d7227: stopping tablet replica
I20260929 21:00:14.509443 7924 tablet_replica.cc:330] T 971c2f439d264794bc0ce7550b6690cc P c5523b2f4ad14d37ba187ff45359077e: stopping tablet replica
I20260929 21:00:14.509582 7923 raft_consensus.cc:2229] T 971c2f439d264794bc0ce7550b6690cc P 527d3ead14f6443b9fef7865f31d7227 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:00:14.509624 7924 raft_consensus.cc:2229] T 971c2f439d264794bc0ce7550b6690cc P c5523b2f4ad14d37ba187ff45359077e [term 1 LEADER]: Raft consensus shutting down.
I20260929 21:00:14.509884 7923 raft_consensus.cc:2258] T 971c2f439d264794bc0ce7550b6690cc P 527d3ead14f6443b9fef7865f31d7227 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:00:14.510031 7924 raft_consensus.cc:2258] T 971c2f439d264794bc0ce7550b6690cc P c5523b2f4ad14d37ba187ff45359077e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:00:14.510031 7737 tablet_service.cc:1514] Processing DeleteTablet for tablet 971c2f439d264794bc0ce7550b6690cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:00:14 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:43792
I20260929 21:00:14.510932 7923 ts_tablet_manager.cc:1905] T 971c2f439d264794bc0ce7550b6690cc P 527d3ead14f6443b9fef7865f31d7227: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:00:14.511312 7924 ts_tablet_manager.cc:1905] T 971c2f439d264794bc0ce7550b6690cc P c5523b2f4ad14d37ba187ff45359077e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:00:14.511590 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 7355
I20260929 21:00:14.515383 7925 tablet_replica.cc:330] T 971c2f439d264794bc0ce7550b6690cc P 34844c02e0f34d5fab45f43e4e5b9956: stopping tablet replica
I20260929 21:00:14.515568 7925 raft_consensus.cc:2229] T 971c2f439d264794bc0ce7550b6690cc P 34844c02e0f34d5fab45f43e4e5b9956 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:00:14.515841 7925 raft_consensus.cc:2258] T 971c2f439d264794bc0ce7550b6690cc P 34844c02e0f34d5fab45f43e4e5b9956 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:00:14.517205 7925 ts_tablet_manager.cc:1905] T 971c2f439d264794bc0ce7550b6690cc P 34844c02e0f34d5fab45f43e4e5b9956: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:00:14.517966 7924 ts_tablet_manager.cc:1918] T 971c2f439d264794bc0ce7550b6690cc P c5523b2f4ad14d37ba187ff45359077e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:00:14.518090 7924 log.cc:1196] T 971c2f439d264794bc0ce7550b6690cc P c5523b2f4ad14d37ba187ff45359077e: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-1/wal/wals/971c2f439d264794bc0ce7550b6690cc
I20260929 21:00:14.518522 7924 ts_tablet_manager.cc:1939] T 971c2f439d264794bc0ce7550b6690cc P c5523b2f4ad14d37ba187ff45359077e: Deleting consensus metadata
I20260929 21:00:14.518818 7923 ts_tablet_manager.cc:1918] T 971c2f439d264794bc0ce7550b6690cc P 527d3ead14f6443b9fef7865f31d7227: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:00:14.518939 7923 log.cc:1196] T 971c2f439d264794bc0ce7550b6690cc P 527d3ead14f6443b9fef7865f31d7227: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-0/wal/wals/971c2f439d264794bc0ce7550b6690cc
I20260929 21:00:14.519380 7923 ts_tablet_manager.cc:1939] T 971c2f439d264794bc0ce7550b6690cc P 527d3ead14f6443b9fef7865f31d7227: Deleting consensus metadata
I20260929 21:00:14.519665 7317 catalog_manager.cc:4841] TS c5523b2f4ad14d37ba187ff45359077e (127.0.97.2:40605): tablet 971c2f439d264794bc0ce7550b6690cc (table test-table [id=776e150f6bad4c2baaff928ee29260b9]) successfully deleted
I20260929 21:00:14.520354 7317 catalog_manager.cc:4841] TS 527d3ead14f6443b9fef7865f31d7227 (127.0.97.1:41637): tablet 971c2f439d264794bc0ce7550b6690cc (table test-table [id=776e150f6bad4c2baaff928ee29260b9]) successfully deleted
I20260929 21:00:14.524865 7925 ts_tablet_manager.cc:1918] T 971c2f439d264794bc0ce7550b6690cc P 34844c02e0f34d5fab45f43e4e5b9956: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:00:14.524998 7925 log.cc:1196] T 971c2f439d264794bc0ce7550b6690cc P 34844c02e0f34d5fab45f43e4e5b9956: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790715556572573-388-0/minicluster-data/ts-2/wal/wals/971c2f439d264794bc0ce7550b6690cc
I20260929 21:00:14.525449 7925 ts_tablet_manager.cc:1939] T 971c2f439d264794bc0ce7550b6690cc P 34844c02e0f34d5fab45f43e4e5b9956: Deleting consensus metadata
I20260929 21:00:14.526571 7319 catalog_manager.cc:4841] TS 34844c02e0f34d5fab45f43e4e5b9956 (127.0.97.3:36861): tablet 971c2f439d264794bc0ce7550b6690cc (table test-table [id=776e150f6bad4c2baaff928ee29260b9]) successfully deleted
I20260929 21:00:14.537712 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.513339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46886 (local address 127.0.97.1:41637)
0929 21:00:14.513510 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:14.513516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:14.523910 (+ 10394us) server_negotiation.cc:407] Connection header received
0929 21:00:14.524155 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:14.524160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:14.524248 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:14.524361 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:14.526218 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:14.527253 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:14.528500 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:14.529164 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:14.529293 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:14.531966 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:14.531984 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:14.532015 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:14.532057 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:14.535117 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:14.535864 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:14.535872 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:14.535875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:14.536008 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:14.536364 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:14.536369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:14.536371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:14.536685 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:14.537001 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:14.537233 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 21:00:14.537389 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 21:00:14.538342 7507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:00:14.596992 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.596504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38557 (local address 127.0.0.1:60846)
0929 21:00:14.596682 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:14.596775 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 21:00:14.598189 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.597867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38557 (local address 127.0.0.1:60850)
0929 21:00:14.598005 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:14.598085 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 21:00:14.636907 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 7355
I20260929 21:00:14.655179 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 7524
I20260929 21:00:14.668140 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.655971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45582 (local address 127.0.97.2:40605)
0929 21:00:14.656147 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:14.656152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:14.656170 (+ 18us) server_negotiation.cc:407] Connection header received
0929 21:00:14.656332 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:14.656336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:14.656404 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:14.656486 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:14.657778 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:14.658449 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:14.659602 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:14.659995 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:14.660083 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:14.662860 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:14.662874 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:14.662891 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:14.662921 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:14.665681 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:14.666325 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:14.666331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:14.666334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:14.666457 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:14.666868 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:14.666871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:14.666873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:14.667196 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:14.667423 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:14.667647 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 21:00:14.667811 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 21:00:14.668784 7655 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:00:14.791761 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 7524
I20260929 21:00:14.814026 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 7673
I20260929 21:00:14.827175 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:14.814716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43806 (local address 127.0.97.3:36861)
0929 21:00:14.814906 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:14.814912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:14.815004 (+ 92us) server_negotiation.cc:407] Connection header received
0929 21:00:14.815217 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:14.815221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:14.815292 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:14.815401 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:14.816661 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:14.817529 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:14.818720 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:14.819104 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:14.819235 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:14.821736 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:14.821749 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:14.821768 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:14.821803 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:14.824451 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:14.825170 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:14.825178 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:14.825180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:14.825304 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:14.825679 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:14.825686 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:14.825689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:14.826045 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:14.826362 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:14.826631 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 21:00:14.826866 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 21:00:14.827859 7797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:00:14.920297 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 7673
I20260929 21:00:14.937093 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 7006
I20260929 21:00:14.938120 7172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:00:15.052708 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 7006
I20260929 21:00:15.072959 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 7106
I20260929 21:00:15.073971 7364 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:00:15.193427 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 7106
2026-09-29T21:00:15Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9534 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 21:00:15 dist-test-slave-1dms krb5kdc[7935](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 21:00:15 dist-test-slave-1dms krb5kdc[7935](info): set up 2 sockets
Sep 29 21:00:15 dist-test-slave-1dms krb5kdc[7935](info): commencing operation
krb5kdc: starting...
W20260929 21:00:16.285835 388 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 21:00:16 dist-test-slave-1dms krb5kdc[7935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T21:00:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:00:16Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260929 21:00:16.498209 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42361
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:00:16.810102 7948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:16.810371 7948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:16.810525 7948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:16.817503 7948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:00:16.817598 7948 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 21:00:16.817687 7948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:16.817798 7948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:16.817879 7948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:00:16.817983 7948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:00:16.827488 7948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42361
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:00:16.828734 7948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:16.830076 7948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:16.841745 7956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:16.841820 7953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:16.841781 7954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:16.842555 7948 server_base.cc:1034] running on GCE node
I20260929 21:00:16.843310 7948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:16.844615 7948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:16.845824 7948 hybrid_clock.cc:648] HybridClock initialized: now 1790715616845742 us; error 53 us; skew 500 ppm
I20260929 21:00:16.846146 7948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:00:16 dist-test-slave-1dms krb5kdc[7935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715616, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:00:16.849224 7948 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260929 21:00:16.850617 7948 webserver.cc:460] Webserver started at http://127.0.97.62:38645/ using document root <none> and password file <none>
I20260929 21:00:16.851292 7948 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:16.851418 7948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:16.851727 7948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:00:16.854547 7948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "2c5be66ba68f48a2a21988e7a837996c"
format_stamp: "Formatted at 2026-09-29 21:00:16 on dist-test-slave-1dms"
server_key: "ea9714c302621c93b08e47cbf45820c7"
server_key_iv: "84bfde6e40fef02db5ecaa95b5a514ba"
server_key_version: "encryptionkey@0"
I20260929 21:00:16.855309 7948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "2c5be66ba68f48a2a21988e7a837996c"
format_stamp: "Formatted at 2026-09-29 21:00:16 on dist-test-slave-1dms"
server_key: "ea9714c302621c93b08e47cbf45820c7"
server_key_iv: "84bfde6e40fef02db5ecaa95b5a514ba"
server_key_version: "encryptionkey@0"
I20260929 21:00:16.860100 7948 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 21:00:16.863324 7963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:16.864548 7948 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 21:00:16.864720 7948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "2c5be66ba68f48a2a21988e7a837996c"
format_stamp: "Formatted at 2026-09-29 21:00:16 on dist-test-slave-1dms"
server_key: "ea9714c302621c93b08e47cbf45820c7"
server_key_iv: "84bfde6e40fef02db5ecaa95b5a514ba"
server_key_version: "encryptionkey@0"
I20260929 21:00:16.865005 7948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:16.879846 7948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:16.883685 7948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:00:16.883951 7948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:16.904826 7948 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42361
I20260929 21:00:16.904878 8024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42361 every 8 connection(s)
I20260929 21:00:16.906180 7948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:00:16.908959 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 7948
I20260929 21:00:16.909165 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:00:16.909999 388 external_mini_cluster.cc:1428] Setting key c0bd3ee9284836b99aa46de1de720aed
I20260929 21:00:16.910385 8025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:00:16.922427 8025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c5be66ba68f48a2a21988e7a837996c: Bootstrap starting.
Sep 29 21:00:16 dist-test-slave-1dms krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 21:00:16.925915 8025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c5be66ba68f48a2a21988e7a837996c: Neither blocks nor log segments found. Creating new log.
I20260929 21:00:16.927083 8025 log.cc:824] T 00000000000000000000000000000000 P 2c5be66ba68f48a2a21988e7a837996c: Log is configured to *not* fsync() on all Append() calls
I20260929 21:00:16.929845 8025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c5be66ba68f48a2a21988e7a837996c: No bootstrap required, opened a new log
I20260929 21:00:16.932201 8028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:16.911995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34994 (local address 127.0.97.62:42361)
0929 21:00:16.912737 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:16.912748 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:16.912787 (+ 39us) server_negotiation.cc:407] Connection header received
0929 21:00:16.914895 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:16.914916 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:16.915303 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:16.915700 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:16.917118 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:16.918369 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:16.919576 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:16.920272 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:16.920486 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:16.924575 (+ 4089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:16.924582 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:16.924596 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:16.924634 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:16.927729 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:16.928447 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:16.928459 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:16.928465 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:16.928615 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:16.929019 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:16.929026 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:16.929029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:16.929504 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:16.929837 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:16.930330 (+ 493us) server_negotiation.cc:299] Negotiation successful
0929 21:00:16.930840 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":347,"threads_started":1}
I20260929 21:00:16.938493 8025 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c5be66ba68f48a2a21988e7a837996c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5be66ba68f48a2a21988e7a837996c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42361 } }
I20260929 21:00:16.938809 8025 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c5be66ba68f48a2a21988e7a837996c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:00:16.938879 8025 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c5be66ba68f48a2a21988e7a837996c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c5be66ba68f48a2a21988e7a837996c, State: Initialized, Role: FOLLOWER
I20260929 21:00:16.939338 8025 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c5be66ba68f48a2a21988e7a837996c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5be66ba68f48a2a21988e7a837996c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42361 } }
I20260929 21:00:16.939527 8025 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c5be66ba68f48a2a21988e7a837996c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:00:16.939680 8025 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c5be66ba68f48a2a21988e7a837996c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:00:16.939845 8025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c5be66ba68f48a2a21988e7a837996c [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:00:16.942166 8025 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c5be66ba68f48a2a21988e7a837996c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5be66ba68f48a2a21988e7a837996c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42361 } }
I20260929 21:00:16.942644 8025 leader_election.cc:302] T 00000000000000000000000000000000 P 2c5be66ba68f48a2a21988e7a837996c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c5be66ba68f48a2a21988e7a837996c; no voters:
I20260929 21:00:16.943270 8025 leader_election.cc:288] T 00000000000000000000000000000000 P 2c5be66ba68f48a2a21988e7a837996c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 21:00:16.943449 8030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c5be66ba68f48a2a21988e7a837996c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:00:16.944165 8030 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c5be66ba68f48a2a21988e7a837996c [term 1 LEADER]: Becoming Leader. State: Replica: 2c5be66ba68f48a2a21988e7a837996c, State: Running, Role: LEADER
I20260929 21:00:16.944686 8030 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c5be66ba68f48a2a21988e7a837996c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5be66ba68f48a2a21988e7a837996c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42361 } }
I20260929 21:00:16.945358 8025 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c5be66ba68f48a2a21988e7a837996c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:00:16.947476 8031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c5be66ba68f48a2a21988e7a837996c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c5be66ba68f48a2a21988e7a837996c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5be66ba68f48a2a21988e7a837996c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42361 } } }
I20260929 21:00:16.947655 8032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c5be66ba68f48a2a21988e7a837996c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c5be66ba68f48a2a21988e7a837996c. Latest consensus state: current_term: 1 leader_uuid: "2c5be66ba68f48a2a21988e7a837996c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5be66ba68f48a2a21988e7a837996c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42361 } } }
I20260929 21:00:16.947857 8031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c5be66ba68f48a2a21988e7a837996c [sys.catalog]: This master's current role is: LEADER
I20260929 21:00:16.947927 8032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c5be66ba68f48a2a21988e7a837996c [sys.catalog]: This master's current role is: LEADER
I20260929 21:00:16.948626 8035 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:00:16.953967 8035 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:00:16.962896 8035 catalog_manager.cc:1347] Generated new cluster ID: 4d78fb59f5924ef08d2ef12d29862ae6
I20260929 21:00:16.963013 8035 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:00:16.972927 8035 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:00:16.974135 8035 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:00:16.985103 8035 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c5be66ba68f48a2a21988e7a837996c: Generated new TSK 0
I20260929 21:00:16.986017 8035 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260929 21:00:17.044389 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42361
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:00:17.364605 8051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:17.364931 8051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:17.365123 8051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:17.372516 8051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:17.372687 8051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:17.372859 8051 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 21:00:17.372990 8051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 21:00:17.382907 8051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:00:17.384195 8051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:17.385567 8051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:17.398375 8056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:17.398319 8059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:17.398362 8057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:17.399799 8051 server_base.cc:1034] running on GCE node
I20260929 21:00:17.400375 8051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:17.401340 8051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:17.402554 8051 hybrid_clock.cc:648] HybridClock initialized: now 1790715617402501 us; error 42 us; skew 500 ppm
I20260929 21:00:17.402842 8051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:00:17 dist-test-slave-1dms krb5kdc[7935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715617, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:00:17.405898 8051 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260929 21:00:17.407289 8051 webserver.cc:460] Webserver started at http://127.0.97.1:40265/ using document root <none> and password file <none>
I20260929 21:00:17.407984 8051 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:17.408114 8051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:17.408430 8051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:00:17.411288 8051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-0/data/instance:
uuid: "20f350f99da445e7b1bf54099a7f7e9a"
format_stamp: "Formatted at 2026-09-29 21:00:17 on dist-test-slave-1dms"
server_key: "69ea34b73d620a1a1a77bb0f8dae0296"
server_key_iv: "7af50938e818d0a64e727dbcf245c0b9"
server_key_version: "encryptionkey@0"
I20260929 21:00:17.412060 8051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-0/wal/instance:
uuid: "20f350f99da445e7b1bf54099a7f7e9a"
format_stamp: "Formatted at 2026-09-29 21:00:17 on dist-test-slave-1dms"
server_key: "69ea34b73d620a1a1a77bb0f8dae0296"
server_key_iv: "7af50938e818d0a64e727dbcf245c0b9"
server_key_version: "encryptionkey@0"
I20260929 21:00:17.416962 8051 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 21:00:17.420274 8066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:17.421655 8051 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:00:17.421829 8051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-0/wal
uuid: "20f350f99da445e7b1bf54099a7f7e9a"
format_stamp: "Formatted at 2026-09-29 21:00:17 on dist-test-slave-1dms"
server_key: "69ea34b73d620a1a1a77bb0f8dae0296"
server_key_iv: "7af50938e818d0a64e727dbcf245c0b9"
server_key_version: "encryptionkey@0"
I20260929 21:00:17.422055 8051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:17.440003 8051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:17.444386 8051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:00:17.444662 8051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:17.447822 8051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:00:17.447957 8051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:17.448120 8051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:00:17.448246 8051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 29 21:00:17 dist-test-slave-1dms krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715617, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 21:00:17.471952 8074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:17.448000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48841 (local address 127.0.97.62:42361)
0929 21:00:17.448790 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:17.448796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:17.449873 (+ 1077us) server_negotiation.cc:407] Connection header received
0929 21:00:17.451973 (+ 2100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:17.451979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:17.452079 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:17.452236 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:17.454567 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:17.455678 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:17.458166 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:17.458878 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:17.459012 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:17.464725 (+ 5713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:17.464730 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:17.464733 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:17.464775 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:17.468200 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:17.469377 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:17.469382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:17.469384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:17.469552 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:17.470123 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:17.470126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:17.470128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:17.470382 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:17.470488 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:17.471526 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0929 21:00:17.471682 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":511,"threads_started":1}
I20260929 21:00:17.474244 8076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:17.448730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42361 (local address 127.0.97.1:48841)
0929 21:00:17.449672 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:17.449720 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:17.451683 (+ 1963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:17.452513 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:17.452524 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:17.453382 (+ 858us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:17.454310 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:17.454342 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:17.455843 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:17.455848 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:17.457961 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:17.457980 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:17.459344 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:17.459354 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:17.459677 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:17.460644 (+ 967us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:17.460701 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:17.464435 (+ 3734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:17.468367 (+ 3932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:17.468377 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:17.468401 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:17.469239 (+ 838us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:17.469676 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:17.469684 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:17.469694 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:17.469997 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:17.470509 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:17.470521 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:17.471087 (+ 566us) client_negotiation.cc:769] Sending connection context
0929 21:00:17.471468 (+ 381us) client_negotiation.cc:240] Negotiation successful
0929 21:00:17.472161 (+ 693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":413,"threads_started":1}
I20260929 21:00:17.501403 8051 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35627
I20260929 21:00:17.501434 8186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35627 every 8 connection(s)
I20260929 21:00:17.502269 8139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:17.485103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42361 (local address 127.0.97.1:38207)
0929 21:00:17.485867 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:17.485883 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:17.486043 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:17.486575 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:17.486578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:17.487273 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:17.487771 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:17.487785 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:17.489428 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:17.489438 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:17.491511 (+ 2073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:17.491529 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:17.492910 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:17.492916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:17.493044 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:17.493751 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:17.493782 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:17.495194 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:17.499377 (+ 4183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:17.499385 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:17.499391 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:17.499873 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:17.500333 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:17.500338 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:17.500342 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:17.500541 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:17.501320 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:17.501324 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:17.501560 (+ 236us) client_negotiation.cc:769] Sending connection context
0929 21:00:17.501723 (+ 163us) client_negotiation.cc:240] Negotiation successful
0929 21:00:17.501961 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":434,"threads_started":1}
I20260929 21:00:17.502543 8074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:17.484690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38207 (local address 127.0.97.62:42361)
0929 21:00:17.484934 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:17.484942 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:17.485972 (+ 1030us) server_negotiation.cc:407] Connection header received
0929 21:00:17.486179 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:17.486186 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:17.486292 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:17.486392 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:17.487976 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:17.489232 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:17.491721 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:17.492480 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:17.492619 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:17.495453 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:17.495461 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:17.495468 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:17.495513 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:17.499188 (+ 3675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:17.500001 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:17.500007 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:17.500011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:17.500191 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:17.500714 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:17.500721 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:17.500726 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:17.501166 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:17.501342 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:17.501834 (+ 492us) server_negotiation.cc:299] Negotiation successful
0929 21:00:17.502068 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 21:00:17.503026 8051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
I20260929 21:00:17.505057 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 8051
I20260929 21:00:17.505244 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-0/wal/instance
I20260929 21:00:17.506021 388 external_mini_cluster.cc:1428] Setting key 43c01e9d17482030305d9125a78428bc
I20260929 21:00:17.505239 8187 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42361
I20260929 21:00:17.506275 8187 heartbeater.cc:464] Registering TS with master...
I20260929 21:00:17.507043 8187 heartbeater.cc:511] Master 127.0.97.62:42361 requested a full tablet report, sending...
I20260929 21:00:17.509099 7980 ts_manager.cc:194] Registered new tserver with Master: 20f350f99da445e7b1bf54099a7f7e9a (127.0.97.1:35627)
I20260929 21:00:17.511189 7980 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48841
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260929 21:00:17.552261 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42361
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:00:17.866839 8191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:17.867105 8191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:17.867201 8191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:17.874397 8191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:17.874496 8191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:17.874578 8191 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 21:00:17.874657 8191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 21:00:17.884068 8191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:00:17.885324 8191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:17.886629 8191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:17.899087 8196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:17.899037 8199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:17.899040 8197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:17.900007 8191 server_base.cc:1034] running on GCE node
I20260929 21:00:17.900522 8191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:17.901417 8191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:17.902621 8191 hybrid_clock.cc:648] HybridClock initialized: now 1790715617902565 us; error 48 us; skew 500 ppm
I20260929 21:00:17.902918 8191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:00:17 dist-test-slave-1dms krb5kdc[7935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715617, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:00:17.910305 8191 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260929 21:00:17.911798 8191 webserver.cc:460] Webserver started at http://127.0.97.2:33917/ using document root <none> and password file <none>
I20260929 21:00:17.912462 8191 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:17.912545 8191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:17.912925 8191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:00:17.915645 8191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-1/data/instance:
uuid: "402c512a26964bd5bc24a332eb2a3860"
format_stamp: "Formatted at 2026-09-29 21:00:17 on dist-test-slave-1dms"
server_key: "9831071ba6627d2a5585a86af2edcab9"
server_key_iv: "e08632691fa67bfdd1e36890ec6cf627"
server_key_version: "encryptionkey@0"
I20260929 21:00:17.916410 8191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-1/wal/instance:
uuid: "402c512a26964bd5bc24a332eb2a3860"
format_stamp: "Formatted at 2026-09-29 21:00:17 on dist-test-slave-1dms"
server_key: "9831071ba6627d2a5585a86af2edcab9"
server_key_iv: "e08632691fa67bfdd1e36890ec6cf627"
server_key_version: "encryptionkey@0"
I20260929 21:00:17.921312 8191 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 21:00:17.924409 8206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:17.925751 8191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:00:17.925925 8191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-1/wal
uuid: "402c512a26964bd5bc24a332eb2a3860"
format_stamp: "Formatted at 2026-09-29 21:00:17 on dist-test-slave-1dms"
server_key: "9831071ba6627d2a5585a86af2edcab9"
server_key_iv: "e08632691fa67bfdd1e36890ec6cf627"
server_key_version: "encryptionkey@0"
I20260929 21:00:17.926129 8191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:17.940217 8191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:17.944682 8191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:00:17.945026 8191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:17.947980 8191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:00:17.948089 8191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:17.948223 8191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:00:17.948278 8191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 21:00:17 dist-test-slave-1dms krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715617, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 21:00:17.970305 8074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:17.948351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38343 (local address 127.0.97.62:42361)
0929 21:00:17.948564 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:17.948573 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:17.950065 (+ 1492us) server_negotiation.cc:407] Connection header received
0929 21:00:17.952174 (+ 2109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:17.952180 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:17.952281 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:17.952392 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:17.954546 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:17.955317 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:17.957056 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:17.957763 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:17.957916 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:17.963241 (+ 5325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:17.963264 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:17.963271 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:17.963315 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:17.966726 (+ 3411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:17.967436 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:17.967441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:17.967445 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:17.967611 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:17.968113 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:17.968118 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:17.968121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:17.968408 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:17.968560 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:17.969525 (+ 965us) server_negotiation.cc:299] Negotiation successful
0929 21:00:17.969802 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 21:00:17.971309 8215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:17.949037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42361 (local address 127.0.97.2:38343)
0929 21:00:17.949908 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:17.949954 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:17.951879 (+ 1925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:17.952696 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:17.952709 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:17.953547 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:17.954369 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:17.954390 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:17.955483 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:17.955487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:17.956913 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:17.956925 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:17.958033 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:17.958044 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:17.958412 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:17.959255 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:17.959301 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:17.962917 (+ 3616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:17.966862 (+ 3945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:17.966871 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:17.966891 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:17.967305 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:17.967745 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:17.967759 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:17.967763 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:17.967987 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:17.968534 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:17.968542 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:17.969000 (+ 458us) client_negotiation.cc:769] Sending connection context
0929 21:00:17.969393 (+ 393us) client_negotiation.cc:240] Negotiation successful
0929 21:00:17.970006 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":339,"threads_started":1}
I20260929 21:00:17.997439 8074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:17.978158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48611 (local address 127.0.97.62:42361)
0929 21:00:17.978338 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:17.978349 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:17.981885 (+ 3536us) server_negotiation.cc:407] Connection header received
0929 21:00:17.982094 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:17.982103 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:17.982209 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:17.982353 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:17.983238 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:17.984706 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:17.986732 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:17.987500 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:17.987647 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:17.990410 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:17.990419 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:17.990427 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:17.990473 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:17.994563 (+ 4090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:17.995213 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:17.995220 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:17.995226 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:17.995424 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:17.995765 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:17.995773 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:17.995778 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:17.996277 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:17.996459 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:17.996822 (+ 363us) server_negotiation.cc:299] Negotiation successful
0929 21:00:17.997105 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 21:00:17.997448 8268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:17.977840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42361 (local address 127.0.97.2:48611)
0929 21:00:17.981751 (+ 3911us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:17.981774 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:17.981947 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:17.982366 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:17.982369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:17.982792 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:17.983100 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:17.983109 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:17.984928 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:17.984935 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:17.986564 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:17.986581 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:17.987700 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:17.987709 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:17.987965 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:17.988702 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:17.988736 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:17.990147 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:17.994700 (+ 4553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:17.994704 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:17.994706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:17.995086 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:17.995536 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:17.995540 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:17.995542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:17.995653 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:17.996417 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:17.996420 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:17.996567 (+ 147us) client_negotiation.cc:769] Sending connection context
0929 21:00:17.996664 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 21:00:17.996988 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3718,"thread_start_us":3550,"threads_started":1}
I20260929 21:00:18.007531 8191 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43105
I20260929 21:00:18.007535 8325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43105 every 8 connection(s)
I20260929 21:00:18.008718 8191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
I20260929 21:00:18.010702 8326 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42361
I20260929 21:00:18.010937 8326 heartbeater.cc:464] Registering TS with master...
I20260929 21:00:18.011552 8326 heartbeater.cc:511] Master 127.0.97.62:42361 requested a full tablet report, sending...
I20260929 21:00:18.012053 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 8191
I20260929 21:00:18.012219 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-1/wal/instance
I20260929 21:00:18.012809 388 external_mini_cluster.cc:1428] Setting key b21b2d318c4857007faf8240d8c7e093
I20260929 21:00:18.013100 7980 ts_manager.cc:194] Registered new tserver with Master: 402c512a26964bd5bc24a332eb2a3860 (127.0.97.2:43105)
I20260929 21:00:18.014070 7980 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38343
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260929 21:00:18.058396 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42361
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:00:18.376757 8330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:18.377118 8330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:18.377269 8330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:18.384562 8330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:18.384684 8330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:18.384881 8330 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 21:00:18.385017 8330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 21:00:18.394901 8330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:00:18.396189 8330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:18.397571 8330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:18.410697 8336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:18.410655 8335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:18.410672 8338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:18.411470 8330 server_base.cc:1034] running on GCE node
I20260929 21:00:18.412051 8330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:18.413043 8330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:18.414269 8330 hybrid_clock.cc:648] HybridClock initialized: now 1790715618414196 us; error 65 us; skew 500 ppm
I20260929 21:00:18.414567 8330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:00:18 dist-test-slave-1dms krb5kdc[7935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715618, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:00:18.417587 8330 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260929 21:00:18.419082 8330 webserver.cc:460] Webserver started at http://127.0.97.3:45293/ using document root <none> and password file <none>
I20260929 21:00:18.419772 8330 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:18.419898 8330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:18.420207 8330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:00:18.423101 8330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-2/data/instance:
uuid: "4407b5bc980b46f88295bed4d965b8a0"
format_stamp: "Formatted at 2026-09-29 21:00:18 on dist-test-slave-1dms"
server_key: "4e548689974d07936330941aa38b1270"
server_key_iv: "52048654e711a30c8f09f72033680417"
server_key_version: "encryptionkey@0"
I20260929 21:00:18.423877 8330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4407b5bc980b46f88295bed4d965b8a0"
format_stamp: "Formatted at 2026-09-29 21:00:18 on dist-test-slave-1dms"
server_key: "4e548689974d07936330941aa38b1270"
server_key_iv: "52048654e711a30c8f09f72033680417"
server_key_version: "encryptionkey@0"
I20260929 21:00:18.429170 8330 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 21:00:18.432586 8345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:18.434000 8330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:00:18.434134 8330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-2/wal
uuid: "4407b5bc980b46f88295bed4d965b8a0"
format_stamp: "Formatted at 2026-09-29 21:00:18 on dist-test-slave-1dms"
server_key: "4e548689974d07936330941aa38b1270"
server_key_iv: "52048654e711a30c8f09f72033680417"
server_key_version: "encryptionkey@0"
I20260929 21:00:18.434315 8330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:18.458359 8330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:18.462975 8330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:00:18.463299 8330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:18.466320 8330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:00:18.466419 8330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:18.466569 8330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:00:18.466638 8330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 21:00:18 dist-test-slave-1dms krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715618, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 21:00:18.489537 8074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:18.468021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47927 (local address 127.0.97.62:42361)
0929 21:00:18.468206 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:18.468212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:18.468524 (+ 312us) server_negotiation.cc:407] Connection header received
0929 21:00:18.470871 (+ 2347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:18.470877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:18.470969 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:18.471079 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:18.473469 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:18.474315 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:18.475930 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:18.476395 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:18.476533 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:18.481575 (+ 5042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:18.481584 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:18.481591 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:18.481635 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:18.485974 (+ 4339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:18.486694 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:18.486703 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:18.486709 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:18.486906 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:18.487388 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:18.487396 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:18.487401 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:18.487846 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:18.488023 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:18.488678 (+ 655us) server_negotiation.cc:299] Negotiation successful
0929 21:00:18.489016 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 21:00:18.489995 8354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:18.467419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42361 (local address 127.0.97.3:47927)
0929 21:00:18.468369 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:18.468406 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:18.470556 (+ 2150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:18.471504 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:18.471512 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:18.472337 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:18.473281 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:18.473302 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:18.474478 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:18.474485 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:18.475768 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:18.475780 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:18.476627 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:18.476634 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:18.476918 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:18.477841 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:18.477883 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:18.481323 (+ 3440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:18.486123 (+ 4800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:18.486136 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:18.486154 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:18.486557 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:18.487045 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:18.487049 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:18.487052 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:18.487268 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:18.487976 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:18.487981 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:18.488303 (+ 322us) client_negotiation.cc:769] Sending connection context
0929 21:00:18.488518 (+ 215us) client_negotiation.cc:240] Negotiation successful
0929 21:00:18.488951 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":448,"threads_started":1}
I20260929 21:00:18.513971 8187 heartbeater.cc:503] Master 127.0.97.62:42361 was elected leader, sending a full tablet report...
I20260929 21:00:18.533613 8074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:18.501115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46741 (local address 127.0.97.62:42361)
0929 21:00:18.506271 (+ 5156us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:18.506280 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:18.517479 (+ 11199us) server_negotiation.cc:407] Connection header received
0929 21:00:18.518006 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:18.518013 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:18.518123 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:18.518228 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:18.519910 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:18.521058 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:18.523212 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:18.523805 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:18.523943 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:18.526761 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:18.526769 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:18.526778 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:18.526808 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:18.530298 (+ 3490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:18.531319 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:18.531323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:18.531325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:18.531476 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:18.531965 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:18.531968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:18.531970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:18.532371 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:18.532536 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:18.533003 (+ 467us) server_negotiation.cc:299] Negotiation successful
0929 21:00:18.533225 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4965}
I20260929 21:00:18.533776 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:18.501103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42361 (local address 127.0.97.3:46741)
0929 21:00:18.517331 (+ 16228us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:18.517355 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:18.517860 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:18.518376 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:18.518381 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:18.519135 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:18.519723 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:18.519740 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:18.521204 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:18.521213 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:18.522983 (+ 1770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:18.522995 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:18.524275 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:18.524286 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:18.524496 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:18.525267 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:18.525297 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:18.526563 (+ 1266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:18.530464 (+ 3901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:18.530471 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:18.530478 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:18.531127 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:18.531631 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:18.531637 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:18.531641 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:18.531817 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:18.532503 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:18.532511 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:18.532698 (+ 187us) client_negotiation.cc:769] Sending connection context
0929 21:00:18.532885 (+ 187us) client_negotiation.cc:240] Negotiation successful
0929 21:00:18.533255 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16026,"thread_start_us":16436,"threads_started":1}
I20260929 21:00:18.537925 8330 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35253
I20260929 21:00:18.537946 8464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35253 every 8 connection(s)
I20260929 21:00:18.539258 8330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
I20260929 21:00:18.541383 8465 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42361
I20260929 21:00:18.541582 8465 heartbeater.cc:464] Registering TS with master...
I20260929 21:00:18.542157 8465 heartbeater.cc:511] Master 127.0.97.62:42361 requested a full tablet report, sending...
I20260929 21:00:18.543349 7979 ts_manager.cc:194] Registered new tserver with Master: 4407b5bc980b46f88295bed4d965b8a0 (127.0.97.3:35253)
I20260929 21:00:18.544126 7979 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47927
I20260929 21:00:18.548381 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 8330
I20260929 21:00:18.548540 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-2/wal/instance
I20260929 21:00:18.549057 388 external_mini_cluster.cc:1428] Setting key 647eaca3bd672db9491abe3089a1385a
I20260929 21:00:18.551857 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 21:00:18 dist-test-slave-1dms krb5kdc[7935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715618, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 21:00:18 dist-test-slave-1dms krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:18 dist-test-slave-1dms krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 21:00:18.598110 8074 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:18.588329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42590 (local address 127.0.97.62:42361)
0929 21:00:18.588505 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:18.588509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:18.588710 (+ 201us) server_negotiation.cc:407] Connection header received
0929 21:00:18.588880 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:18.588884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:18.588947 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:18.589037 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:18.590241 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:18.590968 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:18.592175 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:18.592627 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:18.592761 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:18.597884 (+ 5123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42590: BlockingRecv error: recv got EOF from 127.0.0.1:42590 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 21:00:18 dist-test-slave-1dms krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715618, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 21:00:18.613191 8074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:18.600998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42592 (local address 127.0.97.62:42361)
0929 21:00:18.601138 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:18.601142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:18.601504 (+ 362us) server_negotiation.cc:407] Connection header received
0929 21:00:18.601630 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:18.601633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:18.601699 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:18.601773 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:18.602824 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:18.603471 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:18.604954 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:18.605333 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:18.605422 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:18.608628 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:18.608633 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:18.608636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:18.608665 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:18.611213 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:18.611743 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:18.611747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:18.611749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:18.611872 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:18.612159 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:18.612162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:18.612163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:18.612385 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:18.612574 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:18.612738 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 21:00:18.612966 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 21:00:18.616796 7979 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 21:00:18.623849 7979 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 21:00:18.645444 8485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:18.635771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43105 (local address 127.0.0.1:50516)
0929 21:00:18.637499 (+ 1728us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:18.637520 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:18.637646 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:18.638541 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:18.638553 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:18.638586 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:00:18.639638 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:18.639653 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:18.641175 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:18.641182 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:18.643124 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:18.643136 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:18.644339 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:18.644346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:18.644527 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:18.644572 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 21:00:18.644719 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 21:00:18.644976 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1611,"mutex_wait_us":211,"spinlock_wait_cycles":10368,"thread_start_us":1465,"threads_started":1}
I20260929 21:00:18.646247 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:18.636323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50516 (local address 127.0.97.2:43105)
0929 21:00:18.637062 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:18.637073 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:18.637787 (+ 714us) server_negotiation.cc:407] Connection header received
0929 21:00:18.637893 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:18.637902 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:18.638157 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:18.638339 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:00:18.639880 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:18.641054 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:18.643279 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:18.644135 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:18.644272 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:18.644493 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:18.645761 (+ 1268us) server_negotiation.cc:299] Negotiation successful
0929 21:00:18.645948 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":402,"threads_started":1}
I20260929 21:00:18.647531 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:18.635251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35253 (local address 127.0.0.1:41072)
0929 21:00:18.636160 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:18.636210 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:18.636379 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:18.638589 (+ 2210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:18.638594 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:18.638629 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:00:18.639805 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:18.639818 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:18.641974 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:18.641981 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:18.644910 (+ 2929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:18.644925 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:18.646781 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:18.646807 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:18.646979 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:18.647001 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 21:00:18.647072 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 21:00:18.647155 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":224,"spinlock_wait_cycles":384,"thread_start_us":509,"threads_started":1}
I20260929 21:00:18.647624 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:18.635627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41072 (local address 127.0.97.3:35253)
0929 21:00:18.637979 (+ 2352us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:18.637987 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:18.638013 (+ 26us) server_negotiation.cc:407] Connection header received
0929 21:00:18.638101 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:18.638108 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:18.638316 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:18.638710 (+ 394us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:00:18.640027 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:18.641841 (+ 1814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:18.645088 (+ 3247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:18.646612 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:18.646741 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:18.646961 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:18.647145 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 21:00:18.647299 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2209,"thread_start_us":1997,"threads_started":1}
I20260929 21:00:18.648818 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:18.637248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44256 (local address 127.0.97.1:35627)
0929 21:00:18.638249 (+ 1001us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:18.638260 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:18.638808 (+ 548us) server_negotiation.cc:407] Connection header received
0929 21:00:18.639232 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:18.639240 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:18.639437 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:18.639550 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:00:18.640334 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:18.641755 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:18.645089 (+ 3334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:18.647559 (+ 2470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:18.647691 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:18.647893 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:18.648396 (+ 503us) server_negotiation.cc:299] Negotiation successful
0929 21:00:18.648552 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":840,"thread_start_us":715,"threads_started":1}
I20260929 21:00:18.649619 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:18.636684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35627 (local address 127.0.0.1:44256)
0929 21:00:18.638691 (+ 2007us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:18.638708 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:18.638973 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:18.639726 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:18.639729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:18.639749 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:00:18.640157 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:18.640169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:18.641952 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:18.641962 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:18.644911 (+ 2949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:18.644925 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:18.648029 (+ 3104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:18.648035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:18.648191 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:18.648221 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 21:00:18.649089 (+ 868us) client_negotiation.cc:240] Negotiation successful
0929 21:00:18.649234 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1901,"mutex_wait_us":173,"thread_start_us":1766,"threads_started":1}
I20260929 21:00:18.650053 8256 tablet_service.cc:1467] Processing CreateTablet for tablet 03f7297703c048e99e98dd01297525ed (DEFAULT_TABLE table=test-table [id=68a83227705b483da4ae9d7cd18c6e98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:00:18.651528 8256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03f7297703c048e99e98dd01297525ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:00:18.651079 8395 tablet_service.cc:1467] Processing CreateTablet for tablet 03f7297703c048e99e98dd01297525ed (DEFAULT_TABLE table=test-table [id=68a83227705b483da4ae9d7cd18c6e98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:00:18.652362 8395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03f7297703c048e99e98dd01297525ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:00:18.652583 8117 tablet_service.cc:1467] Processing CreateTablet for tablet 03f7297703c048e99e98dd01297525ed (DEFAULT_TABLE table=test-table [id=68a83227705b483da4ae9d7cd18c6e98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:00:18.653630 8117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03f7297703c048e99e98dd01297525ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:00:18.664328 8489 tablet_bootstrap.cc:492] T 03f7297703c048e99e98dd01297525ed P 20f350f99da445e7b1bf54099a7f7e9a: Bootstrap starting.
I20260929 21:00:18.667794 8489 tablet_bootstrap.cc:654] T 03f7297703c048e99e98dd01297525ed P 20f350f99da445e7b1bf54099a7f7e9a: Neither blocks nor log segments found. Creating new log.
I20260929 21:00:18.669209 8489 log.cc:824] T 03f7297703c048e99e98dd01297525ed P 20f350f99da445e7b1bf54099a7f7e9a: Log is configured to *not* fsync() on all Append() calls
I20260929 21:00:18.671698 8490 tablet_bootstrap.cc:492] T 03f7297703c048e99e98dd01297525ed P 402c512a26964bd5bc24a332eb2a3860: Bootstrap starting.
I20260929 21:00:18.672205 8489 tablet_bootstrap.cc:492] T 03f7297703c048e99e98dd01297525ed P 20f350f99da445e7b1bf54099a7f7e9a: No bootstrap required, opened a new log
I20260929 21:00:18.672492 8489 ts_tablet_manager.cc:1397] T 03f7297703c048e99e98dd01297525ed P 20f350f99da445e7b1bf54099a7f7e9a: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260929 21:00:18.673053 8491 tablet_bootstrap.cc:492] T 03f7297703c048e99e98dd01297525ed P 4407b5bc980b46f88295bed4d965b8a0: Bootstrap starting.
I20260929 21:00:18.675177 8490 tablet_bootstrap.cc:654] T 03f7297703c048e99e98dd01297525ed P 402c512a26964bd5bc24a332eb2a3860: Neither blocks nor log segments found. Creating new log.
I20260929 21:00:18.676380 8491 tablet_bootstrap.cc:654] T 03f7297703c048e99e98dd01297525ed P 4407b5bc980b46f88295bed4d965b8a0: Neither blocks nor log segments found. Creating new log.
I20260929 21:00:18.676530 8490 log.cc:824] T 03f7297703c048e99e98dd01297525ed P 402c512a26964bd5bc24a332eb2a3860: Log is configured to *not* fsync() on all Append() calls
I20260929 21:00:18.677738 8491 log.cc:824] T 03f7297703c048e99e98dd01297525ed P 4407b5bc980b46f88295bed4d965b8a0: Log is configured to *not* fsync() on all Append() calls
I20260929 21:00:18.680590 8491 tablet_bootstrap.cc:492] T 03f7297703c048e99e98dd01297525ed P 4407b5bc980b46f88295bed4d965b8a0: No bootstrap required, opened a new log
I20260929 21:00:18.680936 8491 ts_tablet_manager.cc:1397] T 03f7297703c048e99e98dd01297525ed P 4407b5bc980b46f88295bed4d965b8a0: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20260929 21:00:18.681094 8490 tablet_bootstrap.cc:492] T 03f7297703c048e99e98dd01297525ed P 402c512a26964bd5bc24a332eb2a3860: No bootstrap required, opened a new log
I20260929 21:00:18.681363 8490 ts_tablet_manager.cc:1397] T 03f7297703c048e99e98dd01297525ed P 402c512a26964bd5bc24a332eb2a3860: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260929 21:00:18.681528 8489 raft_consensus.cc:348] T 03f7297703c048e99e98dd01297525ed P 20f350f99da445e7b1bf54099a7f7e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } } peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } }
I20260929 21:00:18.681986 8489 raft_consensus.cc:374] T 03f7297703c048e99e98dd01297525ed P 20f350f99da445e7b1bf54099a7f7e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:00:18.682093 8489 raft_consensus.cc:729] T 03f7297703c048e99e98dd01297525ed P 20f350f99da445e7b1bf54099a7f7e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20f350f99da445e7b1bf54099a7f7e9a, State: Initialized, Role: FOLLOWER
I20260929 21:00:18.682709 8489 consensus_queue.cc:260] T 03f7297703c048e99e98dd01297525ed P 20f350f99da445e7b1bf54099a7f7e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } } peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } }
I20260929 21:00:18.684435 8489 ts_tablet_manager.cc:1428] T 03f7297703c048e99e98dd01297525ed P 20f350f99da445e7b1bf54099a7f7e9a: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260929 21:00:18.689417 8491 raft_consensus.cc:348] T 03f7297703c048e99e98dd01297525ed P 4407b5bc980b46f88295bed4d965b8a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } } peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } }
I20260929 21:00:18.689756 8491 raft_consensus.cc:374] T 03f7297703c048e99e98dd01297525ed P 4407b5bc980b46f88295bed4d965b8a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:00:18.689898 8491 raft_consensus.cc:729] T 03f7297703c048e99e98dd01297525ed P 4407b5bc980b46f88295bed4d965b8a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4407b5bc980b46f88295bed4d965b8a0, State: Initialized, Role: FOLLOWER
I20260929 21:00:18.690490 8491 consensus_queue.cc:260] T 03f7297703c048e99e98dd01297525ed P 4407b5bc980b46f88295bed4d965b8a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } } peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } }
I20260929 21:00:18.691531 8465 heartbeater.cc:503] Master 127.0.97.62:42361 was elected leader, sending a full tablet report...
I20260929 21:00:18.691313 8490 raft_consensus.cc:348] T 03f7297703c048e99e98dd01297525ed P 402c512a26964bd5bc24a332eb2a3860 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } } peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } }
I20260929 21:00:18.691802 8490 raft_consensus.cc:374] T 03f7297703c048e99e98dd01297525ed P 402c512a26964bd5bc24a332eb2a3860 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:00:18.691929 8490 raft_consensus.cc:729] T 03f7297703c048e99e98dd01297525ed P 402c512a26964bd5bc24a332eb2a3860 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 402c512a26964bd5bc24a332eb2a3860, State: Initialized, Role: FOLLOWER
I20260929 21:00:18.692152 8491 ts_tablet_manager.cc:1428] T 03f7297703c048e99e98dd01297525ed P 4407b5bc980b46f88295bed4d965b8a0: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260929 21:00:18.692584 8490 consensus_queue.cc:260] T 03f7297703c048e99e98dd01297525ed P 402c512a26964bd5bc24a332eb2a3860 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } } peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } }
I20260929 21:00:18.693693 8326 heartbeater.cc:503] Master 127.0.97.62:42361 was elected leader, sending a full tablet report...
I20260929 21:00:18.694322 8490 ts_tablet_manager.cc:1428] T 03f7297703c048e99e98dd01297525ed P 402c512a26964bd5bc24a332eb2a3860: Time spent starting tablet: real 0.013s user 0.005s sys 0.007s
W20260929 21:00:18.755919 8188 tablet.cc:2369] T 03f7297703c048e99e98dd01297525ed P 20f350f99da445e7b1bf54099a7f7e9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 21:00:18.760640 8327 tablet.cc:2369] T 03f7297703c048e99e98dd01297525ed P 402c512a26964bd5bc24a332eb2a3860: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 21:00:18.790828 8466 tablet.cc:2369] T 03f7297703c048e99e98dd01297525ed P 4407b5bc980b46f88295bed4d965b8a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 21:00:18.969612 8496 raft_consensus.cc:482] T 03f7297703c048e99e98dd01297525ed P 4407b5bc980b46f88295bed4d965b8a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 21:00:18.969900 8496 raft_consensus.cc:504] T 03f7297703c048e99e98dd01297525ed P 4407b5bc980b46f88295bed4d965b8a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } } peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } }
I20260929 21:00:18.971465 8496 leader_election.cc:288] T 03f7297703c048e99e98dd01297525ed P 4407b5bc980b46f88295bed4d965b8a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20f350f99da445e7b1bf54099a7f7e9a (127.0.97.1:35627), 402c512a26964bd5bc24a332eb2a3860 (127.0.97.2:43105)
I20260929 21:00:18.978209 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:18.971841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42477 (local address 127.0.97.1:35627)
0929 21:00:18.972037 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:18.972043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:18.972280 (+ 237us) server_negotiation.cc:407] Connection header received
0929 21:00:18.972367 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:18.972370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:18.972435 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:18.972536 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:00:18.972986 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:18.974002 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:18.976241 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:18.977396 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:18.977531 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:18.977631 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:18.977835 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 21:00:18.977973 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 21:00:18.978202 8354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:18.971749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35627 (local address 127.0.97.3:42477)
0929 21:00:18.972045 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:18.972065 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:18.972192 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:18.972558 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:18.972562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:18.972588 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:00:18.972793 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:18.972802 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:18.974118 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:18.974122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:18.976097 (+ 1975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:18.976116 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:18.977544 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:18.977548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:18.977681 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:18.977705 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 21:00:18.977785 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 21:00:18.977875 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189}
I20260929 21:00:18.978896 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:18.972031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41875 (local address 127.0.97.2:43105)
0929 21:00:18.972221 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:18.972229 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:18.972676 (+ 447us) server_negotiation.cc:407] Connection header received
0929 21:00:18.972916 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:18.972921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:18.972995 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:18.973091 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:00:18.973578 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:18.974506 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:18.976607 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:18.977862 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:18.977982 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:18.978073 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:18.978489 (+ 416us) server_negotiation.cc:299] Negotiation successful
0929 21:00:18.978634 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 21:00:18.979508 8498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:18.971880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43105 (local address 127.0.97.3:41875)
0929 21:00:18.972568 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:18.972582 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:18.972745 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:18.973108 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:18.973111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:18.973128 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:00:18.973405 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:18.973413 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:18.974660 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:18.974666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:18.976472 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:18.976481 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:18.978117 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:18.978122 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:18.978300 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:18.978330 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 21:00:18.979175 (+ 845us) client_negotiation.cc:240] Negotiation successful
0929 21:00:18.979293 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":407,"threads_started":1}
I20260929 21:00:18.979429 8142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f7297703c048e99e98dd01297525ed" candidate_uuid: "4407b5bc980b46f88295bed4d965b8a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20f350f99da445e7b1bf54099a7f7e9a" is_pre_election: true
I20260929 21:00:18.979995 8142 raft_consensus.cc:2454] T 03f7297703c048e99e98dd01297525ed P 20f350f99da445e7b1bf54099a7f7e9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4407b5bc980b46f88295bed4d965b8a0 in term 0.
I20260929 21:00:18.980523 8281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f7297703c048e99e98dd01297525ed" candidate_uuid: "4407b5bc980b46f88295bed4d965b8a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "402c512a26964bd5bc24a332eb2a3860" is_pre_election: true
I20260929 21:00:18.980630 8348 leader_election.cc:302] T 03f7297703c048e99e98dd01297525ed P 4407b5bc980b46f88295bed4d965b8a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20f350f99da445e7b1bf54099a7f7e9a, 4407b5bc980b46f88295bed4d965b8a0; no voters:
I20260929 21:00:18.981015 8496 raft_consensus.cc:2781] T 03f7297703c048e99e98dd01297525ed P 4407b5bc980b46f88295bed4d965b8a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 21:00:18.981003 8281 raft_consensus.cc:2454] T 03f7297703c048e99e98dd01297525ed P 402c512a26964bd5bc24a332eb2a3860 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4407b5bc980b46f88295bed4d965b8a0 in term 0.
I20260929 21:00:18.981129 8496 raft_consensus.cc:482] T 03f7297703c048e99e98dd01297525ed P 4407b5bc980b46f88295bed4d965b8a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 21:00:18.981243 8496 raft_consensus.cc:3037] T 03f7297703c048e99e98dd01297525ed P 4407b5bc980b46f88295bed4d965b8a0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:00:18.983387 8496 raft_consensus.cc:504] T 03f7297703c048e99e98dd01297525ed P 4407b5bc980b46f88295bed4d965b8a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } } peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } }
I20260929 21:00:18.984009 8496 leader_election.cc:288] T 03f7297703c048e99e98dd01297525ed P 4407b5bc980b46f88295bed4d965b8a0 [CANDIDATE]: Term 1 election: Requested vote from peers 20f350f99da445e7b1bf54099a7f7e9a (127.0.97.1:35627), 402c512a26964bd5bc24a332eb2a3860 (127.0.97.2:43105)
I20260929 21:00:18.984313 8142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f7297703c048e99e98dd01297525ed" candidate_uuid: "4407b5bc980b46f88295bed4d965b8a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20f350f99da445e7b1bf54099a7f7e9a"
I20260929 21:00:18.984496 8142 raft_consensus.cc:3037] T 03f7297703c048e99e98dd01297525ed P 20f350f99da445e7b1bf54099a7f7e9a [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:00:18.984441 8281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f7297703c048e99e98dd01297525ed" candidate_uuid: "4407b5bc980b46f88295bed4d965b8a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "402c512a26964bd5bc24a332eb2a3860"
I20260929 21:00:18.984656 8281 raft_consensus.cc:3037] T 03f7297703c048e99e98dd01297525ed P 402c512a26964bd5bc24a332eb2a3860 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:00:18.986649 8142 raft_consensus.cc:2454] T 03f7297703c048e99e98dd01297525ed P 20f350f99da445e7b1bf54099a7f7e9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4407b5bc980b46f88295bed4d965b8a0 in term 1.
I20260929 21:00:18.986763 8281 raft_consensus.cc:2454] T 03f7297703c048e99e98dd01297525ed P 402c512a26964bd5bc24a332eb2a3860 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4407b5bc980b46f88295bed4d965b8a0 in term 1.
I20260929 21:00:18.987022 8348 leader_election.cc:302] T 03f7297703c048e99e98dd01297525ed P 4407b5bc980b46f88295bed4d965b8a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20f350f99da445e7b1bf54099a7f7e9a, 4407b5bc980b46f88295bed4d965b8a0; no voters:
I20260929 21:00:18.987289 8496 raft_consensus.cc:2781] T 03f7297703c048e99e98dd01297525ed P 4407b5bc980b46f88295bed4d965b8a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:00:18.987658 8496 raft_consensus.cc:686] T 03f7297703c048e99e98dd01297525ed P 4407b5bc980b46f88295bed4d965b8a0 [term 1 LEADER]: Becoming Leader. State: Replica: 4407b5bc980b46f88295bed4d965b8a0, State: Running, Role: LEADER
I20260929 21:00:18.988085 8496 consensus_queue.cc:237] T 03f7297703c048e99e98dd01297525ed P 4407b5bc980b46f88295bed4d965b8a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } } peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } }
I20260929 21:00:18.992154 7980 catalog_manager.cc:5495] T 03f7297703c048e99e98dd01297525ed P 4407b5bc980b46f88295bed4d965b8a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4407b5bc980b46f88295bed4d965b8a0 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "4407b5bc980b46f88295bed4d965b8a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } health_report { overall_health: UNKNOWN } } }
I20260929 21:00:19.011245 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:19.006093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41078 (local address 127.0.97.3:35253)
0929 21:00:19.006300 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:19.006307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:19.006389 (+ 82us) server_negotiation.cc:407] Connection header received
0929 21:00:19.006624 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:19.006628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:19.006704 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:19.006860 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:00:19.007212 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.007895 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.009297 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.009662 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.009791 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:19.010227 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:00:19.010693 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:00:19.010839 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:19.010923 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 21:00:19.010997 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 21:00:19.020035 8281 raft_consensus.cc:1261] T 03f7297703c048e99e98dd01297525ed P 402c512a26964bd5bc24a332eb2a3860 [term 1 FOLLOWER]: Refusing update from remote peer 4407b5bc980b46f88295bed4d965b8a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:00:19.020982 8142 raft_consensus.cc:1261] T 03f7297703c048e99e98dd01297525ed P 20f350f99da445e7b1bf54099a7f7e9a [term 1 FOLLOWER]: Refusing update from remote peer 4407b5bc980b46f88295bed4d965b8a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:00:19.021248 8496 consensus_queue.cc:1035] T 03f7297703c048e99e98dd01297525ed P 4407b5bc980b46f88295bed4d965b8a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:00:19.021734 8499 consensus_queue.cc:1035] T 03f7297703c048e99e98dd01297525ed P 4407b5bc980b46f88295bed4d965b8a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:00:19.046067 7980 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 21:00:19.046324 7980 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 21:00:19.049625 7980 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c5be66ba68f48a2a21988e7a837996c: Sending DeleteTablet for 3 replicas of tablet 03f7297703c048e99e98dd01297525ed
I20260929 21:00:19.050778 8256 tablet_service.cc:1514] Processing DeleteTablet for tablet 03f7297703c048e99e98dd01297525ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:00:19 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50516
I20260929 21:00:19.051013 8395 tablet_service.cc:1514] Processing DeleteTablet for tablet 03f7297703c048e99e98dd01297525ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:00:19 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41072
I20260929 21:00:19.051440 8117 tablet_service.cc:1514] Processing DeleteTablet for tablet 03f7297703c048e99e98dd01297525ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:00:19 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44256
I20260929 21:00:19.052441 7980 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 21:00:19.053498 7980 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 21:00:19.055630 8512 tablet_replica.cc:330] T 03f7297703c048e99e98dd01297525ed P 402c512a26964bd5bc24a332eb2a3860: stopping tablet replica
I20260929 21:00:19.055655 8514 tablet_replica.cc:330] T 03f7297703c048e99e98dd01297525ed P 20f350f99da445e7b1bf54099a7f7e9a: stopping tablet replica
I20260929 21:00:19.056066 8514 raft_consensus.cc:2229] T 03f7297703c048e99e98dd01297525ed P 20f350f99da445e7b1bf54099a7f7e9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:00:19.056066 8512 raft_consensus.cc:2229] T 03f7297703c048e99e98dd01297525ed P 402c512a26964bd5bc24a332eb2a3860 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:00:19.056423 8513 tablet_replica.cc:330] T 03f7297703c048e99e98dd01297525ed P 4407b5bc980b46f88295bed4d965b8a0: stopping tablet replica
I20260929 21:00:19.056782 8512 raft_consensus.cc:2258] T 03f7297703c048e99e98dd01297525ed P 402c512a26964bd5bc24a332eb2a3860 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:00:19.059013 8512 ts_tablet_manager.cc:1905] T 03f7297703c048e99e98dd01297525ed P 402c512a26964bd5bc24a332eb2a3860: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:00:19.060006 8514 raft_consensus.cc:2258] T 03f7297703c048e99e98dd01297525ed P 20f350f99da445e7b1bf54099a7f7e9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:00:19.056823 8513 raft_consensus.cc:2229] T 03f7297703c048e99e98dd01297525ed P 4407b5bc980b46f88295bed4d965b8a0 [term 1 LEADER]: Raft consensus shutting down.
I20260929 21:00:19.061276 8513 raft_consensus.cc:2258] T 03f7297703c048e99e98dd01297525ed P 4407b5bc980b46f88295bed4d965b8a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:00:19.061344 8256 tablet_service.cc:1467] Processing CreateTablet for tablet c7fabce77a454fe5bd12d2e55c93b9e2 (DEFAULT_TABLE table=test-table [id=ffc2dd494f754f119ebb0b3df47039eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:00:19.061882 8256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7fabce77a454fe5bd12d2e55c93b9e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:00:19.061820 8395 tablet_service.cc:1467] Processing CreateTablet for tablet c7fabce77a454fe5bd12d2e55c93b9e2 (DEFAULT_TABLE table=test-table [id=ffc2dd494f754f119ebb0b3df47039eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:00:19.062426 8117 tablet_service.cc:1467] Processing CreateTablet for tablet c7fabce77a454fe5bd12d2e55c93b9e2 (DEFAULT_TABLE table=test-table [id=ffc2dd494f754f119ebb0b3df47039eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:00:19.062839 8117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7fabce77a454fe5bd12d2e55c93b9e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:00:19.063637 8514 ts_tablet_manager.cc:1905] T 03f7297703c048e99e98dd01297525ed P 20f350f99da445e7b1bf54099a7f7e9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:00:19.067848 8512 ts_tablet_manager.cc:1918] T 03f7297703c048e99e98dd01297525ed P 402c512a26964bd5bc24a332eb2a3860: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:00:19.068073 8512 log.cc:1196] T 03f7297703c048e99e98dd01297525ed P 402c512a26964bd5bc24a332eb2a3860: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-1/wal/wals/03f7297703c048e99e98dd01297525ed
I20260929 21:00:19.068130 8395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7fabce77a454fe5bd12d2e55c93b9e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:00:19.068624 8512 ts_tablet_manager.cc:1939] T 03f7297703c048e99e98dd01297525ed P 402c512a26964bd5bc24a332eb2a3860: Deleting consensus metadata
I20260929 21:00:19.069947 8489 tablet_bootstrap.cc:492] T c7fabce77a454fe5bd12d2e55c93b9e2 P 20f350f99da445e7b1bf54099a7f7e9a: Bootstrap starting.
I20260929 21:00:19.071679 8489 tablet_bootstrap.cc:654] T c7fabce77a454fe5bd12d2e55c93b9e2 P 20f350f99da445e7b1bf54099a7f7e9a: Neither blocks nor log segments found. Creating new log.
I20260929 21:00:19.073824 8489 tablet_bootstrap.cc:492] T c7fabce77a454fe5bd12d2e55c93b9e2 P 20f350f99da445e7b1bf54099a7f7e9a: No bootstrap required, opened a new log
I20260929 21:00:19.073957 8491 tablet_bootstrap.cc:492] T c7fabce77a454fe5bd12d2e55c93b9e2 P 4407b5bc980b46f88295bed4d965b8a0: Bootstrap starting.
I20260929 21:00:19.074278 8489 ts_tablet_manager.cc:1397] T c7fabce77a454fe5bd12d2e55c93b9e2 P 20f350f99da445e7b1bf54099a7f7e9a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20260929 21:00:19.074316 8513 ts_tablet_manager.cc:1905] T 03f7297703c048e99e98dd01297525ed P 4407b5bc980b46f88295bed4d965b8a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:00:19.075064 8489 raft_consensus.cc:348] T c7fabce77a454fe5bd12d2e55c93b9e2 P 20f350f99da445e7b1bf54099a7f7e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } }
I20260929 21:00:19.075274 8489 raft_consensus.cc:374] T c7fabce77a454fe5bd12d2e55c93b9e2 P 20f350f99da445e7b1bf54099a7f7e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:00:19.075356 8489 raft_consensus.cc:729] T c7fabce77a454fe5bd12d2e55c93b9e2 P 20f350f99da445e7b1bf54099a7f7e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20f350f99da445e7b1bf54099a7f7e9a, State: Initialized, Role: FOLLOWER
I20260929 21:00:19.075672 8491 tablet_bootstrap.cc:654] T c7fabce77a454fe5bd12d2e55c93b9e2 P 4407b5bc980b46f88295bed4d965b8a0: Neither blocks nor log segments found. Creating new log.
I20260929 21:00:19.075651 8489 consensus_queue.cc:260] T c7fabce77a454fe5bd12d2e55c93b9e2 P 20f350f99da445e7b1bf54099a7f7e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } }
I20260929 21:00:19.076682 8490 tablet_bootstrap.cc:492] T c7fabce77a454fe5bd12d2e55c93b9e2 P 402c512a26964bd5bc24a332eb2a3860: Bootstrap starting.
I20260929 21:00:19.077111 8489 ts_tablet_manager.cc:1428] T c7fabce77a454fe5bd12d2e55c93b9e2 P 20f350f99da445e7b1bf54099a7f7e9a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260929 21:00:19.077666 7964 catalog_manager.cc:4841] TS 402c512a26964bd5bc24a332eb2a3860 (127.0.97.2:43105): tablet 03f7297703c048e99e98dd01297525ed (table test-table [id=68a83227705b483da4ae9d7cd18c6e98]) successfully deleted
I20260929 21:00:19.078297 8490 tablet_bootstrap.cc:654] T c7fabce77a454fe5bd12d2e55c93b9e2 P 402c512a26964bd5bc24a332eb2a3860: Neither blocks nor log segments found. Creating new log.
I20260929 21:00:19.085088 8491 tablet_bootstrap.cc:492] T c7fabce77a454fe5bd12d2e55c93b9e2 P 4407b5bc980b46f88295bed4d965b8a0: No bootstrap required, opened a new log
I20260929 21:00:19.085269 8491 ts_tablet_manager.cc:1397] T c7fabce77a454fe5bd12d2e55c93b9e2 P 4407b5bc980b46f88295bed4d965b8a0: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.001s
I20260929 21:00:19.085376 8513 ts_tablet_manager.cc:1918] T 03f7297703c048e99e98dd01297525ed P 4407b5bc980b46f88295bed4d965b8a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:00:19.085584 8513 log.cc:1196] T 03f7297703c048e99e98dd01297525ed P 4407b5bc980b46f88295bed4d965b8a0: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-2/wal/wals/03f7297703c048e99e98dd01297525ed
I20260929 21:00:19.086091 8513 ts_tablet_manager.cc:1939] T 03f7297703c048e99e98dd01297525ed P 4407b5bc980b46f88295bed4d965b8a0: Deleting consensus metadata
I20260929 21:00:19.086124 8491 raft_consensus.cc:348] T c7fabce77a454fe5bd12d2e55c93b9e2 P 4407b5bc980b46f88295bed4d965b8a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } }
I20260929 21:00:19.086371 8491 raft_consensus.cc:374] T c7fabce77a454fe5bd12d2e55c93b9e2 P 4407b5bc980b46f88295bed4d965b8a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:00:19.086476 8491 raft_consensus.cc:729] T c7fabce77a454fe5bd12d2e55c93b9e2 P 4407b5bc980b46f88295bed4d965b8a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4407b5bc980b46f88295bed4d965b8a0, State: Initialized, Role: FOLLOWER
I20260929 21:00:19.086782 8491 consensus_queue.cc:260] T c7fabce77a454fe5bd12d2e55c93b9e2 P 4407b5bc980b46f88295bed4d965b8a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } }
I20260929 21:00:19.087744 7966 catalog_manager.cc:4841] TS 4407b5bc980b46f88295bed4d965b8a0 (127.0.97.3:35253): tablet 03f7297703c048e99e98dd01297525ed (table test-table [id=68a83227705b483da4ae9d7cd18c6e98]) successfully deleted
I20260929 21:00:19.088869 8514 ts_tablet_manager.cc:1918] T 03f7297703c048e99e98dd01297525ed P 20f350f99da445e7b1bf54099a7f7e9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:00:19.089000 8491 ts_tablet_manager.cc:1428] T c7fabce77a454fe5bd12d2e55c93b9e2 P 4407b5bc980b46f88295bed4d965b8a0: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260929 21:00:19.089030 8514 log.cc:1196] T 03f7297703c048e99e98dd01297525ed P 20f350f99da445e7b1bf54099a7f7e9a: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-0/wal/wals/03f7297703c048e99e98dd01297525ed
I20260929 21:00:19.089546 8514 ts_tablet_manager.cc:1939] T 03f7297703c048e99e98dd01297525ed P 20f350f99da445e7b1bf54099a7f7e9a: Deleting consensus metadata
I20260929 21:00:19.091058 7966 catalog_manager.cc:4841] TS 20f350f99da445e7b1bf54099a7f7e9a (127.0.97.1:35627): tablet 03f7297703c048e99e98dd01297525ed (table test-table [id=68a83227705b483da4ae9d7cd18c6e98]) successfully deleted
I20260929 21:00:19.091833 8490 tablet_bootstrap.cc:492] T c7fabce77a454fe5bd12d2e55c93b9e2 P 402c512a26964bd5bc24a332eb2a3860: No bootstrap required, opened a new log
I20260929 21:00:19.092001 8490 ts_tablet_manager.cc:1397] T c7fabce77a454fe5bd12d2e55c93b9e2 P 402c512a26964bd5bc24a332eb2a3860: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20260929 21:00:19.092798 8490 raft_consensus.cc:348] T c7fabce77a454fe5bd12d2e55c93b9e2 P 402c512a26964bd5bc24a332eb2a3860 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } }
I20260929 21:00:19.093047 8490 raft_consensus.cc:374] T c7fabce77a454fe5bd12d2e55c93b9e2 P 402c512a26964bd5bc24a332eb2a3860 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:00:19.093140 8490 raft_consensus.cc:729] T c7fabce77a454fe5bd12d2e55c93b9e2 P 402c512a26964bd5bc24a332eb2a3860 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 402c512a26964bd5bc24a332eb2a3860, State: Initialized, Role: FOLLOWER
I20260929 21:00:19.093417 8490 consensus_queue.cc:260] T c7fabce77a454fe5bd12d2e55c93b9e2 P 402c512a26964bd5bc24a332eb2a3860 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } }
I20260929 21:00:19.093988 8490 ts_tablet_manager.cc:1428] T c7fabce77a454fe5bd12d2e55c93b9e2 P 402c512a26964bd5bc24a332eb2a3860: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 21:00:19.155304 8499 raft_consensus.cc:482] T c7fabce77a454fe5bd12d2e55c93b9e2 P 4407b5bc980b46f88295bed4d965b8a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 21:00:19.155656 8499 raft_consensus.cc:504] T c7fabce77a454fe5bd12d2e55c93b9e2 P 4407b5bc980b46f88295bed4d965b8a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } }
I20260929 21:00:19.156411 8499 leader_election.cc:288] T c7fabce77a454fe5bd12d2e55c93b9e2 P 4407b5bc980b46f88295bed4d965b8a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20f350f99da445e7b1bf54099a7f7e9a (127.0.97.1:35627), 402c512a26964bd5bc24a332eb2a3860 (127.0.97.2:43105)
I20260929 21:00:19.157037 8142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7fabce77a454fe5bd12d2e55c93b9e2" candidate_uuid: "4407b5bc980b46f88295bed4d965b8a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20f350f99da445e7b1bf54099a7f7e9a" is_pre_election: true
I20260929 21:00:19.157037 8281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7fabce77a454fe5bd12d2e55c93b9e2" candidate_uuid: "4407b5bc980b46f88295bed4d965b8a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "402c512a26964bd5bc24a332eb2a3860" is_pre_election: true
I20260929 21:00:19.157312 8142 raft_consensus.cc:2454] T c7fabce77a454fe5bd12d2e55c93b9e2 P 20f350f99da445e7b1bf54099a7f7e9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4407b5bc980b46f88295bed4d965b8a0 in term 0.
I20260929 21:00:19.157310 8281 raft_consensus.cc:2454] T c7fabce77a454fe5bd12d2e55c93b9e2 P 402c512a26964bd5bc24a332eb2a3860 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4407b5bc980b46f88295bed4d965b8a0 in term 0.
I20260929 21:00:19.157749 8348 leader_election.cc:302] T c7fabce77a454fe5bd12d2e55c93b9e2 P 4407b5bc980b46f88295bed4d965b8a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20f350f99da445e7b1bf54099a7f7e9a, 4407b5bc980b46f88295bed4d965b8a0; no voters:
I20260929 21:00:19.158051 8499 raft_consensus.cc:2781] T c7fabce77a454fe5bd12d2e55c93b9e2 P 4407b5bc980b46f88295bed4d965b8a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 21:00:19.158205 8499 raft_consensus.cc:482] T c7fabce77a454fe5bd12d2e55c93b9e2 P 4407b5bc980b46f88295bed4d965b8a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 21:00:19.158344 8499 raft_consensus.cc:3037] T c7fabce77a454fe5bd12d2e55c93b9e2 P 4407b5bc980b46f88295bed4d965b8a0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:00:19.160728 8499 raft_consensus.cc:504] T c7fabce77a454fe5bd12d2e55c93b9e2 P 4407b5bc980b46f88295bed4d965b8a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } }
I20260929 21:00:19.161429 8499 leader_election.cc:288] T c7fabce77a454fe5bd12d2e55c93b9e2 P 4407b5bc980b46f88295bed4d965b8a0 [CANDIDATE]: Term 1 election: Requested vote from peers 20f350f99da445e7b1bf54099a7f7e9a (127.0.97.1:35627), 402c512a26964bd5bc24a332eb2a3860 (127.0.97.2:43105)
I20260929 21:00:19.161877 8142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7fabce77a454fe5bd12d2e55c93b9e2" candidate_uuid: "4407b5bc980b46f88295bed4d965b8a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20f350f99da445e7b1bf54099a7f7e9a"
I20260929 21:00:19.162003 8281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7fabce77a454fe5bd12d2e55c93b9e2" candidate_uuid: "4407b5bc980b46f88295bed4d965b8a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "402c512a26964bd5bc24a332eb2a3860"
I20260929 21:00:19.162097 8142 raft_consensus.cc:3037] T c7fabce77a454fe5bd12d2e55c93b9e2 P 20f350f99da445e7b1bf54099a7f7e9a [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:00:19.162226 8281 raft_consensus.cc:3037] T c7fabce77a454fe5bd12d2e55c93b9e2 P 402c512a26964bd5bc24a332eb2a3860 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:00:19.164803 8142 raft_consensus.cc:2454] T c7fabce77a454fe5bd12d2e55c93b9e2 P 20f350f99da445e7b1bf54099a7f7e9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4407b5bc980b46f88295bed4d965b8a0 in term 1.
I20260929 21:00:19.165150 8281 raft_consensus.cc:2454] T c7fabce77a454fe5bd12d2e55c93b9e2 P 402c512a26964bd5bc24a332eb2a3860 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4407b5bc980b46f88295bed4d965b8a0 in term 1.
I20260929 21:00:19.165314 8348 leader_election.cc:302] T c7fabce77a454fe5bd12d2e55c93b9e2 P 4407b5bc980b46f88295bed4d965b8a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20f350f99da445e7b1bf54099a7f7e9a, 4407b5bc980b46f88295bed4d965b8a0; no voters:
I20260929 21:00:19.165627 8499 raft_consensus.cc:2781] T c7fabce77a454fe5bd12d2e55c93b9e2 P 4407b5bc980b46f88295bed4d965b8a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:00:19.165899 8499 raft_consensus.cc:686] T c7fabce77a454fe5bd12d2e55c93b9e2 P 4407b5bc980b46f88295bed4d965b8a0 [term 1 LEADER]: Becoming Leader. State: Replica: 4407b5bc980b46f88295bed4d965b8a0, State: Running, Role: LEADER
I20260929 21:00:19.166162 8499 consensus_queue.cc:237] T c7fabce77a454fe5bd12d2e55c93b9e2 P 4407b5bc980b46f88295bed4d965b8a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } }
I20260929 21:00:19.168920 7980 catalog_manager.cc:5495] T c7fabce77a454fe5bd12d2e55c93b9e2 P 4407b5bc980b46f88295bed4d965b8a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4407b5bc980b46f88295bed4d965b8a0 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "4407b5bc980b46f88295bed4d965b8a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } health_report { overall_health: HEALTHY } } }
Sep 29 21:00:19 dist-test-slave-1dms krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715616, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 21:00:19.231411 8524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:19.215052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42361 (local address 127.0.0.1:42594)
0929 21:00:19.215805 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:19.215821 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:19.215933 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:19.216601 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:19.216607 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:19.217492 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:19.217747 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:19.217760 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.218804 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.218810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:19.220344 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:19.220358 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.221231 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.221237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:19.221409 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:19.222177 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:19.222208 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:19.225045 (+ 2837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:19.228937 (+ 3892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:19.228946 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:19.228952 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:19.229358 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:19.229760 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:19.229765 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:19.229769 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:19.229918 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:19.230530 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:19.230536 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:19.230671 (+ 135us) client_negotiation.cc:769] Sending connection context
0929 21:00:19.230892 (+ 221us) client_negotiation.cc:240] Negotiation successful
0929 21:00:19.231109 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":443,"threads_started":1}
I20260929 21:00:19.231751 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:19.215117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42594 (local address 127.0.97.62:42361)
0929 21:00:19.216316 (+ 1199us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:19.216320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:19.216336 (+ 16us) server_negotiation.cc:407] Connection header received
0929 21:00:19.216372 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:19.216375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:19.216420 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:19.216499 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:19.217894 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.218659 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.220514 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.221062 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.221171 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:19.225237 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:19.225244 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:19.225249 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:19.225306 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:19.228761 (+ 3455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:19.229465 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:19.229471 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:19.229474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:19.229612 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:19.230024 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:19.230030 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:19.230033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:19.230417 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:19.230568 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:19.231012 (+ 444us) server_negotiation.cc:299] Negotiation successful
0929 21:00:19.231282 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":1097,"thread_start_us":990,"threads_started":1}
I20260929 21:00:19.236923 7980 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42594:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 21:00:19.238097 7980 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 21:00:19.244746 8117 tablet_service.cc:1467] Processing CreateTablet for tablet 2c63e8382b854d579a597ec40e2ebbe9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=93f845335c374ad8a6e31e0dcf9709ca]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 21:00:19.245029 8256 tablet_service.cc:1467] Processing CreateTablet for tablet 2c63e8382b854d579a597ec40e2ebbe9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=93f845335c374ad8a6e31e0dcf9709ca]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 21:00:19.245352 8117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c63e8382b854d579a597ec40e2ebbe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:00:19.245383 8395 tablet_service.cc:1467] Processing CreateTablet for tablet 2c63e8382b854d579a597ec40e2ebbe9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=93f845335c374ad8a6e31e0dcf9709ca]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 21:00:19.245574 8256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c63e8382b854d579a597ec40e2ebbe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:00:19.245908 8395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c63e8382b854d579a597ec40e2ebbe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:00:19.252723 8489 tablet_bootstrap.cc:492] T 2c63e8382b854d579a597ec40e2ebbe9 P 20f350f99da445e7b1bf54099a7f7e9a: Bootstrap starting.
I20260929 21:00:19.252859 8490 tablet_bootstrap.cc:492] T 2c63e8382b854d579a597ec40e2ebbe9 P 402c512a26964bd5bc24a332eb2a3860: Bootstrap starting.
I20260929 21:00:19.254412 8489 tablet_bootstrap.cc:654] T 2c63e8382b854d579a597ec40e2ebbe9 P 20f350f99da445e7b1bf54099a7f7e9a: Neither blocks nor log segments found. Creating new log.
I20260929 21:00:19.254490 8490 tablet_bootstrap.cc:654] T 2c63e8382b854d579a597ec40e2ebbe9 P 402c512a26964bd5bc24a332eb2a3860: Neither blocks nor log segments found. Creating new log.
I20260929 21:00:19.255414 8491 tablet_bootstrap.cc:492] T 2c63e8382b854d579a597ec40e2ebbe9 P 4407b5bc980b46f88295bed4d965b8a0: Bootstrap starting.
I20260929 21:00:19.257119 8491 tablet_bootstrap.cc:654] T 2c63e8382b854d579a597ec40e2ebbe9 P 4407b5bc980b46f88295bed4d965b8a0: Neither blocks nor log segments found. Creating new log.
I20260929 21:00:19.258620 8490 tablet_bootstrap.cc:492] T 2c63e8382b854d579a597ec40e2ebbe9 P 402c512a26964bd5bc24a332eb2a3860: No bootstrap required, opened a new log
I20260929 21:00:19.258790 8490 ts_tablet_manager.cc:1397] T 2c63e8382b854d579a597ec40e2ebbe9 P 402c512a26964bd5bc24a332eb2a3860: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260929 21:00:19.259541 8490 raft_consensus.cc:348] T 2c63e8382b854d579a597ec40e2ebbe9 P 402c512a26964bd5bc24a332eb2a3860 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } }
I20260929 21:00:19.259796 8490 raft_consensus.cc:374] T 2c63e8382b854d579a597ec40e2ebbe9 P 402c512a26964bd5bc24a332eb2a3860 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:00:19.259882 8490 raft_consensus.cc:729] T 2c63e8382b854d579a597ec40e2ebbe9 P 402c512a26964bd5bc24a332eb2a3860 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 402c512a26964bd5bc24a332eb2a3860, State: Initialized, Role: FOLLOWER
I20260929 21:00:19.260251 8489 tablet_bootstrap.cc:492] T 2c63e8382b854d579a597ec40e2ebbe9 P 20f350f99da445e7b1bf54099a7f7e9a: No bootstrap required, opened a new log
I20260929 21:00:19.260160 8490 consensus_queue.cc:260] T 2c63e8382b854d579a597ec40e2ebbe9 P 402c512a26964bd5bc24a332eb2a3860 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } }
I20260929 21:00:19.260535 8489 ts_tablet_manager.cc:1397] T 2c63e8382b854d579a597ec40e2ebbe9 P 20f350f99da445e7b1bf54099a7f7e9a: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260929 21:00:19.260604 8497 tablet_replica.cc:439] T 2c63e8382b854d579a597ec40e2ebbe9 P 402c512a26964bd5bc24a332eb2a3860: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } } }
I20260929 21:00:19.260968 8497 tablet_replica.cc:442] T 2c63e8382b854d579a597ec40e2ebbe9 P 402c512a26964bd5bc24a332eb2a3860: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 21:00:19.261452 8489 raft_consensus.cc:348] T 2c63e8382b854d579a597ec40e2ebbe9 P 20f350f99da445e7b1bf54099a7f7e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } }
I20260929 21:00:19.261760 8489 raft_consensus.cc:374] T 2c63e8382b854d579a597ec40e2ebbe9 P 20f350f99da445e7b1bf54099a7f7e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:00:19.261897 8489 raft_consensus.cc:729] T 2c63e8382b854d579a597ec40e2ebbe9 P 20f350f99da445e7b1bf54099a7f7e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20f350f99da445e7b1bf54099a7f7e9a, State: Initialized, Role: FOLLOWER
I20260929 21:00:19.262194 8489 consensus_queue.cc:260] T 2c63e8382b854d579a597ec40e2ebbe9 P 20f350f99da445e7b1bf54099a7f7e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } }
I20260929 21:00:19.262741 8495 tablet_replica.cc:439] T 2c63e8382b854d579a597ec40e2ebbe9 P 20f350f99da445e7b1bf54099a7f7e9a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } } }
I20260929 21:00:19.263067 8495 tablet_replica.cc:442] T 2c63e8382b854d579a597ec40e2ebbe9 P 20f350f99da445e7b1bf54099a7f7e9a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 21:00:19.263783 8490 ts_tablet_manager.cc:1428] T 2c63e8382b854d579a597ec40e2ebbe9 P 402c512a26964bd5bc24a332eb2a3860: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260929 21:00:19.266156 8489 ts_tablet_manager.cc:1428] T 2c63e8382b854d579a597ec40e2ebbe9 P 20f350f99da445e7b1bf54099a7f7e9a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260929 21:00:19.267706 8491 tablet_bootstrap.cc:492] T 2c63e8382b854d579a597ec40e2ebbe9 P 4407b5bc980b46f88295bed4d965b8a0: No bootstrap required, opened a new log
I20260929 21:00:19.267907 8491 ts_tablet_manager.cc:1397] T 2c63e8382b854d579a597ec40e2ebbe9 P 4407b5bc980b46f88295bed4d965b8a0: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260929 21:00:19.268641 8491 raft_consensus.cc:348] T 2c63e8382b854d579a597ec40e2ebbe9 P 4407b5bc980b46f88295bed4d965b8a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } }
I20260929 21:00:19.268890 8491 raft_consensus.cc:374] T 2c63e8382b854d579a597ec40e2ebbe9 P 4407b5bc980b46f88295bed4d965b8a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:00:19.268980 8491 raft_consensus.cc:729] T 2c63e8382b854d579a597ec40e2ebbe9 P 4407b5bc980b46f88295bed4d965b8a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4407b5bc980b46f88295bed4d965b8a0, State: Initialized, Role: FOLLOWER
I20260929 21:00:19.269281 8491 consensus_queue.cc:260] T 2c63e8382b854d579a597ec40e2ebbe9 P 4407b5bc980b46f88295bed4d965b8a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } }
I20260929 21:00:19.269750 8499 tablet_replica.cc:439] T 2c63e8382b854d579a597ec40e2ebbe9 P 4407b5bc980b46f88295bed4d965b8a0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } } }
I20260929 21:00:19.270053 8499 tablet_replica.cc:442] T 2c63e8382b854d579a597ec40e2ebbe9 P 4407b5bc980b46f88295bed4d965b8a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 21:00:19.270525 8491 ts_tablet_manager.cc:1428] T 2c63e8382b854d579a597ec40e2ebbe9 P 4407b5bc980b46f88295bed4d965b8a0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 21:00:19.291343 8495 raft_consensus.cc:482] T 2c63e8382b854d579a597ec40e2ebbe9 P 20f350f99da445e7b1bf54099a7f7e9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 21:00:19.291581 8495 raft_consensus.cc:504] T 2c63e8382b854d579a597ec40e2ebbe9 P 20f350f99da445e7b1bf54099a7f7e9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } }
I20260929 21:00:19.293260 8495 leader_election.cc:288] T 2c63e8382b854d579a597ec40e2ebbe9 P 20f350f99da445e7b1bf54099a7f7e9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 402c512a26964bd5bc24a332eb2a3860 (127.0.97.2:43105), 4407b5bc980b46f88295bed4d965b8a0 (127.0.97.3:35253)
I20260929 21:00:19.302469 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:19.293702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48435 (local address 127.0.97.2:43105)
0929 21:00:19.293919 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:19.293929 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:19.294952 (+ 1023us) server_negotiation.cc:407] Connection header received
0929 21:00:19.295276 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:19.295285 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:19.295374 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:19.295483 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:00:19.296490 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.297550 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.300177 (+ 2627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.301454 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.301587 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:19.301686 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:19.302026 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 21:00:19.302174 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 21:00:19.302502 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:19.293528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43105 (local address 127.0.97.1:48435)
0929 21:00:19.294829 (+ 1301us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:19.294852 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:19.295069 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:19.295510 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:19.295516 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:19.295540 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:00:19.296305 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:19.296319 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.297735 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.297740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:19.299988 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:19.300002 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.301667 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.301674 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:19.301849 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:19.301886 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 21:00:19.301985 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 21:00:19.302115 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1135,"thread_start_us":1279,"threads_started":1}
I20260929 21:00:19.303162 8529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:19.293610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35253 (local address 127.0.97.1:48121)
0929 21:00:19.295100 (+ 1490us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:19.295116 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:19.295270 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:19.296708 (+ 1438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:19.296712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:19.296739 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:00:19.297118 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:19.297129 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.299252 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.299258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:19.300890 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:19.300909 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.302458 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.302466 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:19.302653 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:19.302682 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 21:00:19.302786 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 21:00:19.302921 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1380,"thread_start_us":908,"threads_started":1}
I20260929 21:00:19.303280 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:19.294106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48121 (local address 127.0.97.3:35253)
0929 21:00:19.294343 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:19.294351 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:19.295182 (+ 831us) server_negotiation.cc:407] Connection header received
0929 21:00:19.295410 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:19.295414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:19.295532 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:19.295628 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:00:19.297622 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.299089 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.301049 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.302091 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.302208 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:19.302312 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:19.302823 (+ 511us) server_negotiation.cc:299] Negotiation successful
0929 21:00:19.302973 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 21:00:19.304105 8420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c63e8382b854d579a597ec40e2ebbe9" candidate_uuid: "20f350f99da445e7b1bf54099a7f7e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4407b5bc980b46f88295bed4d965b8a0" is_pre_election: true
I20260929 21:00:19.304226 8281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c63e8382b854d579a597ec40e2ebbe9" candidate_uuid: "20f350f99da445e7b1bf54099a7f7e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "402c512a26964bd5bc24a332eb2a3860" is_pre_election: true
I20260929 21:00:19.304487 8281 raft_consensus.cc:2454] T 2c63e8382b854d579a597ec40e2ebbe9 P 402c512a26964bd5bc24a332eb2a3860 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20f350f99da445e7b1bf54099a7f7e9a in term 0.
I20260929 21:00:19.304589 8420 raft_consensus.cc:2454] T 2c63e8382b854d579a597ec40e2ebbe9 P 4407b5bc980b46f88295bed4d965b8a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20f350f99da445e7b1bf54099a7f7e9a in term 0.
I20260929 21:00:19.304989 8067 leader_election.cc:302] T 2c63e8382b854d579a597ec40e2ebbe9 P 20f350f99da445e7b1bf54099a7f7e9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20f350f99da445e7b1bf54099a7f7e9a, 402c512a26964bd5bc24a332eb2a3860; no voters:
I20260929 21:00:19.305413 8495 raft_consensus.cc:2781] T 2c63e8382b854d579a597ec40e2ebbe9 P 20f350f99da445e7b1bf54099a7f7e9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 21:00:19.305548 8495 raft_consensus.cc:482] T 2c63e8382b854d579a597ec40e2ebbe9 P 20f350f99da445e7b1bf54099a7f7e9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 21:00:19.305655 8495 raft_consensus.cc:3037] T 2c63e8382b854d579a597ec40e2ebbe9 P 20f350f99da445e7b1bf54099a7f7e9a [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:00:19.307942 8495 raft_consensus.cc:504] T 2c63e8382b854d579a597ec40e2ebbe9 P 20f350f99da445e7b1bf54099a7f7e9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } }
I20260929 21:00:19.308528 8495 leader_election.cc:288] T 2c63e8382b854d579a597ec40e2ebbe9 P 20f350f99da445e7b1bf54099a7f7e9a [CANDIDATE]: Term 1 election: Requested vote from peers 402c512a26964bd5bc24a332eb2a3860 (127.0.97.2:43105), 4407b5bc980b46f88295bed4d965b8a0 (127.0.97.3:35253)
I20260929 21:00:19.308982 8281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c63e8382b854d579a597ec40e2ebbe9" candidate_uuid: "20f350f99da445e7b1bf54099a7f7e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "402c512a26964bd5bc24a332eb2a3860"
I20260929 21:00:19.309069 8420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c63e8382b854d579a597ec40e2ebbe9" candidate_uuid: "20f350f99da445e7b1bf54099a7f7e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4407b5bc980b46f88295bed4d965b8a0"
I20260929 21:00:19.309219 8281 raft_consensus.cc:3037] T 2c63e8382b854d579a597ec40e2ebbe9 P 402c512a26964bd5bc24a332eb2a3860 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:00:19.309268 8420 raft_consensus.cc:3037] T 2c63e8382b854d579a597ec40e2ebbe9 P 4407b5bc980b46f88295bed4d965b8a0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:00:19.312182 8281 raft_consensus.cc:2454] T 2c63e8382b854d579a597ec40e2ebbe9 P 402c512a26964bd5bc24a332eb2a3860 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20f350f99da445e7b1bf54099a7f7e9a in term 1.
I20260929 21:00:19.312561 8420 raft_consensus.cc:2454] T 2c63e8382b854d579a597ec40e2ebbe9 P 4407b5bc980b46f88295bed4d965b8a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20f350f99da445e7b1bf54099a7f7e9a in term 1.
I20260929 21:00:19.312605 8067 leader_election.cc:302] T 2c63e8382b854d579a597ec40e2ebbe9 P 20f350f99da445e7b1bf54099a7f7e9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20f350f99da445e7b1bf54099a7f7e9a, 402c512a26964bd5bc24a332eb2a3860; no voters:
I20260929 21:00:19.312950 8495 raft_consensus.cc:2781] T 2c63e8382b854d579a597ec40e2ebbe9 P 20f350f99da445e7b1bf54099a7f7e9a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:00:19.313467 8495 raft_consensus.cc:686] T 2c63e8382b854d579a597ec40e2ebbe9 P 20f350f99da445e7b1bf54099a7f7e9a [term 1 LEADER]: Becoming Leader. State: Replica: 20f350f99da445e7b1bf54099a7f7e9a, State: Running, Role: LEADER
I20260929 21:00:19.313851 8495 consensus_queue.cc:237] T 2c63e8382b854d579a597ec40e2ebbe9 P 20f350f99da445e7b1bf54099a7f7e9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } }
I20260929 21:00:19.315860 8531 tablet_replica.cc:439] T 2c63e8382b854d579a597ec40e2ebbe9 P 20f350f99da445e7b1bf54099a7f7e9a: TxnStatusTablet state changed. Reason: New leader 20f350f99da445e7b1bf54099a7f7e9a. Latest consensus state: current_term: 1 leader_uuid: "20f350f99da445e7b1bf54099a7f7e9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } } }
I20260929 21:00:19.316255 8531 tablet_replica.cc:442] T 2c63e8382b854d579a597ec40e2ebbe9 P 20f350f99da445e7b1bf54099a7f7e9a: This TxnStatusTablet replica's current role is: LEADER
I20260929 21:00:19.317020 8533 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 21:00:19.317225 8533 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 21:00:19.317844 7978 catalog_manager.cc:5495] T 2c63e8382b854d579a597ec40e2ebbe9 P 20f350f99da445e7b1bf54099a7f7e9a reported cstate change: term changed from 0 to 1, leader changed from <none> to 20f350f99da445e7b1bf54099a7f7e9a (127.0.97.1). New cstate: current_term: 1 leader_uuid: "20f350f99da445e7b1bf54099a7f7e9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } health_report { overall_health: UNKNOWN } } }
I20260929 21:00:19.354190 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:19.346796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44266 (local address 127.0.97.1:35627)
0929 21:00:19.346970 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:19.346977 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:19.347300 (+ 323us) server_negotiation.cc:407] Connection header received
0929 21:00:19.347590 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:19.347596 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:19.347714 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:19.347876 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:00:19.348279 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.349478 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.351384 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.352029 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.352180 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:19.352754 (+ 574us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:00:19.353493 (+ 739us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:00:19.353634 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:19.353757 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 21:00:19.353845 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 21:00:19.354200 8524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:19.346831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35627 (local address 127.0.0.1:44266)
0929 21:00:19.347181 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:19.347201 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:19.347359 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:19.347909 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:19.347914 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:19.347928 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 21:00:19.348120 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:19.348131 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.349619 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.349625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:19.351225 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:19.351240 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.352201 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.352207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:19.352451 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:19.352519 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 21:00:19.353599 (+ 1080us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 21:00:19.353621 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 21:00:19.353720 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 21:00:19.353880 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185}
I20260929 21:00:19.359050 8281 raft_consensus.cc:1261] T 2c63e8382b854d579a597ec40e2ebbe9 P 402c512a26964bd5bc24a332eb2a3860 [term 1 FOLLOWER]: Refusing update from remote peer 20f350f99da445e7b1bf54099a7f7e9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:00:19.359184 8420 raft_consensus.cc:1261] T 2c63e8382b854d579a597ec40e2ebbe9 P 4407b5bc980b46f88295bed4d965b8a0 [term 1 FOLLOWER]: Refusing update from remote peer 20f350f99da445e7b1bf54099a7f7e9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:00:19.359802 8495 consensus_queue.cc:1035] T 2c63e8382b854d579a597ec40e2ebbe9 P 20f350f99da445e7b1bf54099a7f7e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:00:19.360428 8495 consensus_queue.cc:1035] T 2c63e8382b854d579a597ec40e2ebbe9 P 20f350f99da445e7b1bf54099a7f7e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:00:19.366833 8497 tablet_replica.cc:439] T 2c63e8382b854d579a597ec40e2ebbe9 P 402c512a26964bd5bc24a332eb2a3860: TxnStatusTablet state changed. Reason: New leader 20f350f99da445e7b1bf54099a7f7e9a. Latest consensus state: current_term: 1 leader_uuid: "20f350f99da445e7b1bf54099a7f7e9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } } }
I20260929 21:00:19.367106 8497 tablet_replica.cc:442] T 2c63e8382b854d579a597ec40e2ebbe9 P 402c512a26964bd5bc24a332eb2a3860: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 21:00:19.369263 8531 tablet_replica.cc:439] T 2c63e8382b854d579a597ec40e2ebbe9 P 20f350f99da445e7b1bf54099a7f7e9a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "20f350f99da445e7b1bf54099a7f7e9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } } }
I20260929 21:00:19.369510 8531 tablet_replica.cc:442] T 2c63e8382b854d579a597ec40e2ebbe9 P 20f350f99da445e7b1bf54099a7f7e9a: This TxnStatusTablet replica's current role is: LEADER
I20260929 21:00:19.370008 8503 mvcc.cc:205] Tried to move back new op lower bound from 7334771176887103488 to 7334771176713199616. Current Snapshot: MvccSnapshot[applied={T|T < 7334771176887103488}]
I20260929 21:00:19.371120 8499 tablet_replica.cc:439] T 2c63e8382b854d579a597ec40e2ebbe9 P 4407b5bc980b46f88295bed4d965b8a0: TxnStatusTablet state changed. Reason: New leader 20f350f99da445e7b1bf54099a7f7e9a. Latest consensus state: current_term: 1 leader_uuid: "20f350f99da445e7b1bf54099a7f7e9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } } }
I20260929 21:00:19.371423 8499 tablet_replica.cc:442] T 2c63e8382b854d579a597ec40e2ebbe9 P 4407b5bc980b46f88295bed4d965b8a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 21:00:19.373031 8495 tablet_replica.cc:439] T 2c63e8382b854d579a597ec40e2ebbe9 P 20f350f99da445e7b1bf54099a7f7e9a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "20f350f99da445e7b1bf54099a7f7e9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } } }
I20260929 21:00:19.373390 8495 tablet_replica.cc:442] T 2c63e8382b854d579a597ec40e2ebbe9 P 20f350f99da445e7b1bf54099a7f7e9a: This TxnStatusTablet replica's current role is: LEADER
I20260929 21:00:19.374608 8497 tablet_replica.cc:439] T 2c63e8382b854d579a597ec40e2ebbe9 P 402c512a26964bd5bc24a332eb2a3860: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "20f350f99da445e7b1bf54099a7f7e9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } } }
I20260929 21:00:19.374828 8497 tablet_replica.cc:442] T 2c63e8382b854d579a597ec40e2ebbe9 P 402c512a26964bd5bc24a332eb2a3860: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 21:00:19.379705 8502 mvcc.cc:205] Tried to move back new op lower bound from 7334771176887103488 to 7334771176713199616. Current Snapshot: MvccSnapshot[applied={T|T < 7334771176887103488}]
I20260929 21:00:19.380096 8499 tablet_replica.cc:439] T 2c63e8382b854d579a597ec40e2ebbe9 P 4407b5bc980b46f88295bed4d965b8a0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "20f350f99da445e7b1bf54099a7f7e9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } } }
I20260929 21:00:19.380357 8499 tablet_replica.cc:442] T 2c63e8382b854d579a597ec40e2ebbe9 P 4407b5bc980b46f88295bed4d965b8a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 21:00:19.400969 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:19.393115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53489 (local address 127.0.97.1:35627)
0929 21:00:19.393441 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:19.393449 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:19.393918 (+ 469us) server_negotiation.cc:407] Connection header received
0929 21:00:19.394115 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:19.394121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:19.394223 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:19.394319 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:00:19.394956 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.396162 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.398594 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.399278 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.399392 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:19.400142 (+ 750us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:00:19.400275 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:00:19.400345 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:19.400565 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 21:00:19.400644 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198}
I20260929 21:00:19.401721 8540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:19.393052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35627 (local address 127.0.97.3:53489)
0929 21:00:19.393805 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:19.393825 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:19.393977 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:19.394374 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:19.394379 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:19.394396 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 21:00:19.394790 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:19.394803 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.396328 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.396333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:19.398371 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:19.398386 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.399673 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.399679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:19.399869 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:19.399944 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 21:00:19.400424 (+ 480us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 21:00:19.400433 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 21:00:19.401145 (+ 712us) client_negotiation.cc:240] Negotiation successful
0929 21:00:19.401306 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":424,"threads_started":1}
I20260929 21:00:19.412926 8142 raft_consensus.cc:1261] T c7fabce77a454fe5bd12d2e55c93b9e2 P 20f350f99da445e7b1bf54099a7f7e9a [term 1 FOLLOWER]: Refusing update from remote peer 4407b5bc980b46f88295bed4d965b8a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:00:19.413852 8496 consensus_queue.cc:1035] T c7fabce77a454fe5bd12d2e55c93b9e2 P 4407b5bc980b46f88295bed4d965b8a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:00:19.417023 8281 raft_consensus.cc:1261] T c7fabce77a454fe5bd12d2e55c93b9e2 P 402c512a26964bd5bc24a332eb2a3860 [term 1 FOLLOWER]: Refusing update from remote peer 4407b5bc980b46f88295bed4d965b8a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:00:19.417734 8496 consensus_queue.cc:1035] T c7fabce77a454fe5bd12d2e55c93b9e2 P 4407b5bc980b46f88295bed4d965b8a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:00:19.481369 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:19.463330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49743 (local address 127.0.97.3:35253)
0929 21:00:19.463513 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:19.463519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:19.474337 (+ 10818us) server_negotiation.cc:407] Connection header received
0929 21:00:19.474554 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:19.474560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:19.474642 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:19.474703 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:00:19.475479 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.476707 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.479033 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.479721 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.479852 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:19.480517 (+ 665us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:00:19.480657 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:00:19.480718 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:19.480951 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 21:00:19.481038 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 21:00:19.482231 8545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:19.462972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35253 (local address 127.0.97.1:49743)
0929 21:00:19.474201 (+ 11229us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:19.474222 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:19.474388 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:19.474903 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:19.474907 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:19.474920 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 21:00:19.475297 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:19.475313 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.477008 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.477017 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:19.478871 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:19.478886 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.480017 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.480025 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:19.480223 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:19.480314 (+ 91us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 21:00:19.480794 (+ 480us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 21:00:19.480805 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 21:00:19.481612 (+ 807us) client_negotiation.cc:240] Negotiation successful
0929 21:00:19.481781 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10955,"thread_start_us":10837,"threads_started":1}
I20260929 21:00:19.528793 7978 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 21:00:19.529114 7978 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 21:00:19.532078 7978 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c5be66ba68f48a2a21988e7a837996c: Sending DeleteTablet for 3 replicas of tablet c7fabce77a454fe5bd12d2e55c93b9e2
I20260929 21:00:19.532994 8256 tablet_service.cc:1514] Processing DeleteTablet for tablet c7fabce77a454fe5bd12d2e55c93b9e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:00:19 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50516
I20260929 21:00:19.533372 8117 tablet_service.cc:1514] Processing DeleteTablet for tablet c7fabce77a454fe5bd12d2e55c93b9e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:00:19 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44256
I20260929 21:00:19.533644 8514 tablet_replica.cc:330] T c7fabce77a454fe5bd12d2e55c93b9e2 P 20f350f99da445e7b1bf54099a7f7e9a: stopping tablet replica
I20260929 21:00:19.533993 8514 raft_consensus.cc:2229] T c7fabce77a454fe5bd12d2e55c93b9e2 P 20f350f99da445e7b1bf54099a7f7e9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:00:19.534199 8514 raft_consensus.cc:2258] T c7fabce77a454fe5bd12d2e55c93b9e2 P 20f350f99da445e7b1bf54099a7f7e9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:00:19.534999 8395 tablet_service.cc:1514] Processing DeleteTablet for tablet c7fabce77a454fe5bd12d2e55c93b9e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:00:19 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41072
I20260929 21:00:19.535272 8513 tablet_replica.cc:330] T c7fabce77a454fe5bd12d2e55c93b9e2 P 4407b5bc980b46f88295bed4d965b8a0: stopping tablet replica
I20260929 21:00:19.535495 8513 raft_consensus.cc:2229] T c7fabce77a454fe5bd12d2e55c93b9e2 P 4407b5bc980b46f88295bed4d965b8a0 [term 1 LEADER]: Raft consensus shutting down.
I20260929 21:00:19.535843 8513 raft_consensus.cc:2258] T c7fabce77a454fe5bd12d2e55c93b9e2 P 4407b5bc980b46f88295bed4d965b8a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:00:19.536309 8512 tablet_replica.cc:330] T c7fabce77a454fe5bd12d2e55c93b9e2 P 402c512a26964bd5bc24a332eb2a3860: stopping tablet replica
I20260929 21:00:19.536501 8512 raft_consensus.cc:2229] T c7fabce77a454fe5bd12d2e55c93b9e2 P 402c512a26964bd5bc24a332eb2a3860 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:00:19.536691 8512 raft_consensus.cc:2258] T c7fabce77a454fe5bd12d2e55c93b9e2 P 402c512a26964bd5bc24a332eb2a3860 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:00:19.537103 8514 ts_tablet_manager.cc:1905] T c7fabce77a454fe5bd12d2e55c93b9e2 P 20f350f99da445e7b1bf54099a7f7e9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:00:19.537918 8512 ts_tablet_manager.cc:1905] T c7fabce77a454fe5bd12d2e55c93b9e2 P 402c512a26964bd5bc24a332eb2a3860: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:00:19.544427 8514 ts_tablet_manager.cc:1918] T c7fabce77a454fe5bd12d2e55c93b9e2 P 20f350f99da445e7b1bf54099a7f7e9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 21:00:19.544427 8512 ts_tablet_manager.cc:1918] T c7fabce77a454fe5bd12d2e55c93b9e2 P 402c512a26964bd5bc24a332eb2a3860: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 21:00:19.544571 8512 log.cc:1196] T c7fabce77a454fe5bd12d2e55c93b9e2 P 402c512a26964bd5bc24a332eb2a3860: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-1/wal/wals/c7fabce77a454fe5bd12d2e55c93b9e2
I20260929 21:00:19.544710 8514 log.cc:1196] T c7fabce77a454fe5bd12d2e55c93b9e2 P 20f350f99da445e7b1bf54099a7f7e9a: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-0/wal/wals/c7fabce77a454fe5bd12d2e55c93b9e2
I20260929 21:00:19.545230 8514 ts_tablet_manager.cc:1939] T c7fabce77a454fe5bd12d2e55c93b9e2 P 20f350f99da445e7b1bf54099a7f7e9a: Deleting consensus metadata
I20260929 21:00:19.545852 8513 ts_tablet_manager.cc:1905] T c7fabce77a454fe5bd12d2e55c93b9e2 P 4407b5bc980b46f88295bed4d965b8a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:00:19.553020 7966 catalog_manager.cc:4841] TS 20f350f99da445e7b1bf54099a7f7e9a (127.0.97.1:35627): tablet c7fabce77a454fe5bd12d2e55c93b9e2 (table test-table [id=ffc2dd494f754f119ebb0b3df47039eb]) successfully deleted
I20260929 21:00:19.561489 8512 ts_tablet_manager.cc:1939] T c7fabce77a454fe5bd12d2e55c93b9e2 P 402c512a26964bd5bc24a332eb2a3860: Deleting consensus metadata
I20260929 21:00:19.562628 7964 catalog_manager.cc:4841] TS 402c512a26964bd5bc24a332eb2a3860 (127.0.97.2:43105): tablet c7fabce77a454fe5bd12d2e55c93b9e2 (table test-table [id=ffc2dd494f754f119ebb0b3df47039eb]) successfully deleted
I20260929 21:00:19.563036 8513 ts_tablet_manager.cc:1918] T c7fabce77a454fe5bd12d2e55c93b9e2 P 4407b5bc980b46f88295bed4d965b8a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 21:00:19.563211 8513 log.cc:1196] T c7fabce77a454fe5bd12d2e55c93b9e2 P 4407b5bc980b46f88295bed4d965b8a0: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-2/wal/wals/c7fabce77a454fe5bd12d2e55c93b9e2
I20260929 21:00:19.563655 8513 ts_tablet_manager.cc:1939] T c7fabce77a454fe5bd12d2e55c93b9e2 P 4407b5bc980b46f88295bed4d965b8a0: Deleting consensus metadata
I20260929 21:00:19.564656 7966 catalog_manager.cc:4841] TS 4407b5bc980b46f88295bed4d965b8a0 (127.0.97.3:35253): tablet c7fabce77a454fe5bd12d2e55c93b9e2 (table test-table [id=ffc2dd494f754f119ebb0b3df47039eb]) successfully deleted
Sep 29 21:00:19 dist-test-slave-1dms krb5kdc[7935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 21:00:19 dist-test-slave-1dms krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:19 dist-test-slave-1dms krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 21:00:19.622799 8525 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:19.611708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42596 (local address 127.0.97.62:42361)
0929 21:00:19.611932 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:19.611939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:19.612331 (+ 392us) server_negotiation.cc:407] Connection header received
0929 21:00:19.612470 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:19.612476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:19.612560 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:19.612672 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:19.613980 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.615124 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.616516 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.617196 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.617316 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:19.622528 (+ 5212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42596: BlockingRecv error: recv got EOF from 127.0.0.1:42596 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 21:00:19 dist-test-slave-1dms krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 21:00:19.644964 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:19.625773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42606 (local address 127.0.97.62:42361)
0929 21:00:19.625951 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:19.625957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:19.629380 (+ 3423us) server_negotiation.cc:407] Connection header received
0929 21:00:19.629585 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:19.629589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:19.629652 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:19.629749 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:19.631279 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.632082 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.633922 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.634360 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.634483 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:19.639169 (+ 4686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:19.639174 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:19.639178 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:19.639219 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:19.642421 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:19.643042 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:19.643047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:19.643050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:19.643202 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:19.643552 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:19.643557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:19.643560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:19.643888 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:19.644121 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:19.644371 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 21:00:19.644572 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 21:00:19.648623 7978 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 21:00:19.649433 7978 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 21:00:19.656529 8117 tablet_service.cc:1467] Processing CreateTablet for tablet b3e742f6ba3142f89cbfa57e43324f49 (DEFAULT_TABLE table=test-table [id=9ec8f98795ee4f28b5f5a06f6de52e49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:00:19.656827 8256 tablet_service.cc:1467] Processing CreateTablet for tablet b3e742f6ba3142f89cbfa57e43324f49 (DEFAULT_TABLE table=test-table [id=9ec8f98795ee4f28b5f5a06f6de52e49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:00:19.657044 8117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3e742f6ba3142f89cbfa57e43324f49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:00:19.657299 8256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3e742f6ba3142f89cbfa57e43324f49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:00:19.661916 8395 tablet_service.cc:1467] Processing CreateTablet for tablet b3e742f6ba3142f89cbfa57e43324f49 (DEFAULT_TABLE table=test-table [id=9ec8f98795ee4f28b5f5a06f6de52e49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:00:19.662130 8490 tablet_bootstrap.cc:492] T b3e742f6ba3142f89cbfa57e43324f49 P 402c512a26964bd5bc24a332eb2a3860: Bootstrap starting.
I20260929 21:00:19.662354 8395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3e742f6ba3142f89cbfa57e43324f49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:00:19.663859 8490 tablet_bootstrap.cc:654] T b3e742f6ba3142f89cbfa57e43324f49 P 402c512a26964bd5bc24a332eb2a3860: Neither blocks nor log segments found. Creating new log.
I20260929 21:00:19.664422 8489 tablet_bootstrap.cc:492] T b3e742f6ba3142f89cbfa57e43324f49 P 20f350f99da445e7b1bf54099a7f7e9a: Bootstrap starting.
I20260929 21:00:19.666165 8489 tablet_bootstrap.cc:654] T b3e742f6ba3142f89cbfa57e43324f49 P 20f350f99da445e7b1bf54099a7f7e9a: Neither blocks nor log segments found. Creating new log.
I20260929 21:00:19.674146 8491 tablet_bootstrap.cc:492] T b3e742f6ba3142f89cbfa57e43324f49 P 4407b5bc980b46f88295bed4d965b8a0: Bootstrap starting.
I20260929 21:00:19.674575 8490 tablet_bootstrap.cc:492] T b3e742f6ba3142f89cbfa57e43324f49 P 402c512a26964bd5bc24a332eb2a3860: No bootstrap required, opened a new log
I20260929 21:00:19.674708 8490 ts_tablet_manager.cc:1397] T b3e742f6ba3142f89cbfa57e43324f49 P 402c512a26964bd5bc24a332eb2a3860: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.009s
I20260929 21:00:19.674892 8489 tablet_bootstrap.cc:492] T b3e742f6ba3142f89cbfa57e43324f49 P 20f350f99da445e7b1bf54099a7f7e9a: No bootstrap required, opened a new log
I20260929 21:00:19.675064 8489 ts_tablet_manager.cc:1397] T b3e742f6ba3142f89cbfa57e43324f49 P 20f350f99da445e7b1bf54099a7f7e9a: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260929 21:00:19.675469 8490 raft_consensus.cc:348] T b3e742f6ba3142f89cbfa57e43324f49 P 402c512a26964bd5bc24a332eb2a3860 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } }
I20260929 21:00:19.675721 8490 raft_consensus.cc:374] T b3e742f6ba3142f89cbfa57e43324f49 P 402c512a26964bd5bc24a332eb2a3860 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:00:19.675812 8490 raft_consensus.cc:729] T b3e742f6ba3142f89cbfa57e43324f49 P 402c512a26964bd5bc24a332eb2a3860 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 402c512a26964bd5bc24a332eb2a3860, State: Initialized, Role: FOLLOWER
I20260929 21:00:19.675947 8491 tablet_bootstrap.cc:654] T b3e742f6ba3142f89cbfa57e43324f49 P 4407b5bc980b46f88295bed4d965b8a0: Neither blocks nor log segments found. Creating new log.
I20260929 21:00:19.675935 8489 raft_consensus.cc:348] T b3e742f6ba3142f89cbfa57e43324f49 P 20f350f99da445e7b1bf54099a7f7e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } }
I20260929 21:00:19.676056 8490 consensus_queue.cc:260] T b3e742f6ba3142f89cbfa57e43324f49 P 402c512a26964bd5bc24a332eb2a3860 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } }
I20260929 21:00:19.676283 8489 raft_consensus.cc:374] T b3e742f6ba3142f89cbfa57e43324f49 P 20f350f99da445e7b1bf54099a7f7e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:00:19.676406 8489 raft_consensus.cc:729] T b3e742f6ba3142f89cbfa57e43324f49 P 20f350f99da445e7b1bf54099a7f7e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20f350f99da445e7b1bf54099a7f7e9a, State: Initialized, Role: FOLLOWER
I20260929 21:00:19.676656 8489 consensus_queue.cc:260] T b3e742f6ba3142f89cbfa57e43324f49 P 20f350f99da445e7b1bf54099a7f7e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } }
I20260929 21:00:19.677044 8541 raft_consensus.cc:482] T b3e742f6ba3142f89cbfa57e43324f49 P 20f350f99da445e7b1bf54099a7f7e9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 21:00:19.677309 8489 ts_tablet_manager.cc:1428] T b3e742f6ba3142f89cbfa57e43324f49 P 20f350f99da445e7b1bf54099a7f7e9a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 21:00:19.677227 8541 raft_consensus.cc:504] T b3e742f6ba3142f89cbfa57e43324f49 P 20f350f99da445e7b1bf54099a7f7e9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } }
I20260929 21:00:19.678007 8541 leader_election.cc:288] T b3e742f6ba3142f89cbfa57e43324f49 P 20f350f99da445e7b1bf54099a7f7e9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 402c512a26964bd5bc24a332eb2a3860 (127.0.97.2:43105), 4407b5bc980b46f88295bed4d965b8a0 (127.0.97.3:35253)
I20260929 21:00:19.678988 8490 ts_tablet_manager.cc:1428] T b3e742f6ba3142f89cbfa57e43324f49 P 402c512a26964bd5bc24a332eb2a3860: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260929 21:00:19.678925 8281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3e742f6ba3142f89cbfa57e43324f49" candidate_uuid: "20f350f99da445e7b1bf54099a7f7e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "402c512a26964bd5bc24a332eb2a3860" is_pre_election: true
I20260929 21:00:19.679150 8281 raft_consensus.cc:2454] T b3e742f6ba3142f89cbfa57e43324f49 P 402c512a26964bd5bc24a332eb2a3860 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20f350f99da445e7b1bf54099a7f7e9a in term 0.
I20260929 21:00:19.679616 8067 leader_election.cc:302] T b3e742f6ba3142f89cbfa57e43324f49 P 20f350f99da445e7b1bf54099a7f7e9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20f350f99da445e7b1bf54099a7f7e9a, 402c512a26964bd5bc24a332eb2a3860; no voters:
I20260929 21:00:19.679764 8420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3e742f6ba3142f89cbfa57e43324f49" candidate_uuid: "20f350f99da445e7b1bf54099a7f7e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4407b5bc980b46f88295bed4d965b8a0" is_pre_election: true
I20260929 21:00:19.680365 8531 raft_consensus.cc:2781] T b3e742f6ba3142f89cbfa57e43324f49 P 20f350f99da445e7b1bf54099a7f7e9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 21:00:19.680445 8531 raft_consensus.cc:482] T b3e742f6ba3142f89cbfa57e43324f49 P 20f350f99da445e7b1bf54099a7f7e9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 21:00:19.680523 8531 raft_consensus.cc:3037] T b3e742f6ba3142f89cbfa57e43324f49 P 20f350f99da445e7b1bf54099a7f7e9a [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:00:19.682716 8531 raft_consensus.cc:504] T b3e742f6ba3142f89cbfa57e43324f49 P 20f350f99da445e7b1bf54099a7f7e9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } }
I20260929 21:00:19.683305 8531 leader_election.cc:288] T b3e742f6ba3142f89cbfa57e43324f49 P 20f350f99da445e7b1bf54099a7f7e9a [CANDIDATE]: Term 1 election: Requested vote from peers 402c512a26964bd5bc24a332eb2a3860 (127.0.97.2:43105), 4407b5bc980b46f88295bed4d965b8a0 (127.0.97.3:35253)
W20260929 21:00:19.683637 8069 leader_election.cc:341] T b3e742f6ba3142f89cbfa57e43324f49 P 20f350f99da445e7b1bf54099a7f7e9a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4407b5bc980b46f88295bed4d965b8a0 (127.0.97.3:35253): Illegal state: must be running to vote when last-logged opid is not known
I20260929 21:00:19.683713 8281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3e742f6ba3142f89cbfa57e43324f49" candidate_uuid: "20f350f99da445e7b1bf54099a7f7e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "402c512a26964bd5bc24a332eb2a3860"
I20260929 21:00:19.683894 8281 raft_consensus.cc:3037] T b3e742f6ba3142f89cbfa57e43324f49 P 402c512a26964bd5bc24a332eb2a3860 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:00:19.684242 8420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3e742f6ba3142f89cbfa57e43324f49" candidate_uuid: "20f350f99da445e7b1bf54099a7f7e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4407b5bc980b46f88295bed4d965b8a0"
W20260929 21:00:19.684751 8069 leader_election.cc:341] T b3e742f6ba3142f89cbfa57e43324f49 P 20f350f99da445e7b1bf54099a7f7e9a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4407b5bc980b46f88295bed4d965b8a0 (127.0.97.3:35253): Illegal state: must be running to vote when last-logged opid is not known
I20260929 21:00:19.687808 8281 raft_consensus.cc:2454] T b3e742f6ba3142f89cbfa57e43324f49 P 402c512a26964bd5bc24a332eb2a3860 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20f350f99da445e7b1bf54099a7f7e9a in term 1.
I20260929 21:00:19.688910 8067 leader_election.cc:302] T b3e742f6ba3142f89cbfa57e43324f49 P 20f350f99da445e7b1bf54099a7f7e9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20f350f99da445e7b1bf54099a7f7e9a, 402c512a26964bd5bc24a332eb2a3860; no voters: 4407b5bc980b46f88295bed4d965b8a0
I20260929 21:00:19.689407 8531 raft_consensus.cc:2781] T b3e742f6ba3142f89cbfa57e43324f49 P 20f350f99da445e7b1bf54099a7f7e9a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:00:19.689642 8531 raft_consensus.cc:686] T b3e742f6ba3142f89cbfa57e43324f49 P 20f350f99da445e7b1bf54099a7f7e9a [term 1 LEADER]: Becoming Leader. State: Replica: 20f350f99da445e7b1bf54099a7f7e9a, State: Running, Role: LEADER
I20260929 21:00:19.689962 8531 consensus_queue.cc:237] T b3e742f6ba3142f89cbfa57e43324f49 P 20f350f99da445e7b1bf54099a7f7e9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } }
I20260929 21:00:19.690486 8491 tablet_bootstrap.cc:492] T b3e742f6ba3142f89cbfa57e43324f49 P 4407b5bc980b46f88295bed4d965b8a0: No bootstrap required, opened a new log
I20260929 21:00:19.690646 8491 ts_tablet_manager.cc:1397] T b3e742f6ba3142f89cbfa57e43324f49 P 4407b5bc980b46f88295bed4d965b8a0: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.002s
I20260929 21:00:19.691583 8491 raft_consensus.cc:348] T b3e742f6ba3142f89cbfa57e43324f49 P 4407b5bc980b46f88295bed4d965b8a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } }
I20260929 21:00:19.691915 8491 raft_consensus.cc:374] T b3e742f6ba3142f89cbfa57e43324f49 P 4407b5bc980b46f88295bed4d965b8a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:00:19.692035 8491 raft_consensus.cc:729] T b3e742f6ba3142f89cbfa57e43324f49 P 4407b5bc980b46f88295bed4d965b8a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4407b5bc980b46f88295bed4d965b8a0, State: Initialized, Role: FOLLOWER
I20260929 21:00:19.692284 8491 consensus_queue.cc:260] T b3e742f6ba3142f89cbfa57e43324f49 P 4407b5bc980b46f88295bed4d965b8a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } }
I20260929 21:00:19.693043 8491 ts_tablet_manager.cc:1428] T b3e742f6ba3142f89cbfa57e43324f49 P 4407b5bc980b46f88295bed4d965b8a0: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260929 21:00:19.693053 7978 catalog_manager.cc:5495] T b3e742f6ba3142f89cbfa57e43324f49 P 20f350f99da445e7b1bf54099a7f7e9a reported cstate change: term changed from 0 to 1, leader changed from <none> to 20f350f99da445e7b1bf54099a7f7e9a (127.0.97.1). New cstate: current_term: 1 leader_uuid: "20f350f99da445e7b1bf54099a7f7e9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } health_report { overall_health: UNKNOWN } } }
I20260929 21:00:19.708596 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:19.702994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44270 (local address 127.0.97.1:35627)
0929 21:00:19.703183 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:19.703189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:19.703306 (+ 117us) server_negotiation.cc:407] Connection header received
0929 21:00:19.703521 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:19.703524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:19.703582 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:19.703653 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:00:19.704060 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.704795 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.706779 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.707247 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.707337 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:19.707790 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:00:19.707899 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:00:19.707961 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:19.708182 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 21:00:19.708245 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 21:00:19.711589 8420 raft_consensus.cc:3037] T b3e742f6ba3142f89cbfa57e43324f49 P 4407b5bc980b46f88295bed4d965b8a0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:00:19.711625 8281 raft_consensus.cc:1261] T b3e742f6ba3142f89cbfa57e43324f49 P 402c512a26964bd5bc24a332eb2a3860 [term 1 FOLLOWER]: Refusing update from remote peer 20f350f99da445e7b1bf54099a7f7e9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:00:19.712250 8531 consensus_queue.cc:1035] T b3e742f6ba3142f89cbfa57e43324f49 P 20f350f99da445e7b1bf54099a7f7e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:00:19.714846 8420 raft_consensus.cc:1261] T b3e742f6ba3142f89cbfa57e43324f49 P 4407b5bc980b46f88295bed4d965b8a0 [term 1 FOLLOWER]: Refusing update from remote peer 20f350f99da445e7b1bf54099a7f7e9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:00:19.715512 8531 consensus_queue.cc:1035] T b3e742f6ba3142f89cbfa57e43324f49 P 20f350f99da445e7b1bf54099a7f7e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:00:19.718717 8504 mvcc.cc:205] Tried to move back new op lower bound from 7334771178333802496 to 7334771178253266944. Current Snapshot: MvccSnapshot[applied={T|T < 7334771178333802496 or (T in {7334771178333802496})}]
I20260929 21:00:19.727909 7978 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 21:00:19.728224 7978 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 21:00:19.732316 7978 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c5be66ba68f48a2a21988e7a837996c: Sending DeleteTablet for 3 replicas of tablet b3e742f6ba3142f89cbfa57e43324f49
I20260929 21:00:19.733058 8117 tablet_service.cc:1514] Processing DeleteTablet for tablet b3e742f6ba3142f89cbfa57e43324f49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:00:19 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44256
I20260929 21:00:19.733363 8514 tablet_replica.cc:330] T b3e742f6ba3142f89cbfa57e43324f49 P 20f350f99da445e7b1bf54099a7f7e9a: stopping tablet replica
I20260929 21:00:19.733475 8256 tablet_service.cc:1514] Processing DeleteTablet for tablet b3e742f6ba3142f89cbfa57e43324f49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:00:19 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50516
I20260929 21:00:19.733549 8514 raft_consensus.cc:2229] T b3e742f6ba3142f89cbfa57e43324f49 P 20f350f99da445e7b1bf54099a7f7e9a [term 1 LEADER]: Raft consensus shutting down.
I20260929 21:00:19.733919 8512 tablet_replica.cc:330] T b3e742f6ba3142f89cbfa57e43324f49 P 402c512a26964bd5bc24a332eb2a3860: stopping tablet replica
I20260929 21:00:19.733940 8395 tablet_service.cc:1514] Processing DeleteTablet for tablet b3e742f6ba3142f89cbfa57e43324f49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:00:19 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41072
I20260929 21:00:19.734056 8514 raft_consensus.cc:2258] T b3e742f6ba3142f89cbfa57e43324f49 P 20f350f99da445e7b1bf54099a7f7e9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:00:19.734104 8512 raft_consensus.cc:2229] T b3e742f6ba3142f89cbfa57e43324f49 P 402c512a26964bd5bc24a332eb2a3860 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:00:19.734195 8513 tablet_replica.cc:330] T b3e742f6ba3142f89cbfa57e43324f49 P 4407b5bc980b46f88295bed4d965b8a0: stopping tablet replica
I20260929 21:00:19.734329 8512 raft_consensus.cc:2258] T b3e742f6ba3142f89cbfa57e43324f49 P 402c512a26964bd5bc24a332eb2a3860 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:00:19.734356 8513 raft_consensus.cc:2229] T b3e742f6ba3142f89cbfa57e43324f49 P 4407b5bc980b46f88295bed4d965b8a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:00:19.734556 8513 raft_consensus.cc:2258] T b3e742f6ba3142f89cbfa57e43324f49 P 4407b5bc980b46f88295bed4d965b8a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:00:19.735698 8513 ts_tablet_manager.cc:1905] T b3e742f6ba3142f89cbfa57e43324f49 P 4407b5bc980b46f88295bed4d965b8a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:00:19.735869 7980 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 21:00:19.736159 8514 ts_tablet_manager.cc:1905] T b3e742f6ba3142f89cbfa57e43324f49 P 20f350f99da445e7b1bf54099a7f7e9a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 21:00:19.736497 7980 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 21:00:19.740728 8512 ts_tablet_manager.cc:1905] T b3e742f6ba3142f89cbfa57e43324f49 P 402c512a26964bd5bc24a332eb2a3860: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:00:19.742426 8513 ts_tablet_manager.cc:1918] T b3e742f6ba3142f89cbfa57e43324f49 P 4407b5bc980b46f88295bed4d965b8a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:00:19.742560 8513 log.cc:1196] T b3e742f6ba3142f89cbfa57e43324f49 P 4407b5bc980b46f88295bed4d965b8a0: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-2/wal/wals/b3e742f6ba3142f89cbfa57e43324f49
I20260929 21:00:19.742964 8513 ts_tablet_manager.cc:1939] T b3e742f6ba3142f89cbfa57e43324f49 P 4407b5bc980b46f88295bed4d965b8a0: Deleting consensus metadata
I20260929 21:00:19.743971 7966 catalog_manager.cc:4841] TS 4407b5bc980b46f88295bed4d965b8a0 (127.0.97.3:35253): tablet b3e742f6ba3142f89cbfa57e43324f49 (table test-table [id=9ec8f98795ee4f28b5f5a06f6de52e49]) successfully deleted
I20260929 21:00:19.746948 8256 tablet_service.cc:1467] Processing CreateTablet for tablet 6ca4fd9cb957431b81257eb461d0b408 (DEFAULT_TABLE table=test-table [id=6149f5fdfb6e42cfbdf387998c06602b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:00:19.746946 8117 tablet_service.cc:1467] Processing CreateTablet for tablet 6ca4fd9cb957431b81257eb461d0b408 (DEFAULT_TABLE table=test-table [id=6149f5fdfb6e42cfbdf387998c06602b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:00:19.747367 8256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ca4fd9cb957431b81257eb461d0b408. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:00:19.747381 8117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ca4fd9cb957431b81257eb461d0b408. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:00:19.750358 8514 ts_tablet_manager.cc:1918] T b3e742f6ba3142f89cbfa57e43324f49 P 20f350f99da445e7b1bf54099a7f7e9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:00:19.752650 8512 ts_tablet_manager.cc:1918] T b3e742f6ba3142f89cbfa57e43324f49 P 402c512a26964bd5bc24a332eb2a3860: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:00:19.752875 8512 log.cc:1196] T b3e742f6ba3142f89cbfa57e43324f49 P 402c512a26964bd5bc24a332eb2a3860: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-1/wal/wals/b3e742f6ba3142f89cbfa57e43324f49
I20260929 21:00:19.753618 8512 ts_tablet_manager.cc:1939] T b3e742f6ba3142f89cbfa57e43324f49 P 402c512a26964bd5bc24a332eb2a3860: Deleting consensus metadata
I20260929 21:00:19.754058 8490 tablet_bootstrap.cc:492] T 6ca4fd9cb957431b81257eb461d0b408 P 402c512a26964bd5bc24a332eb2a3860: Bootstrap starting.
I20260929 21:00:19.754747 8395 tablet_service.cc:1467] Processing CreateTablet for tablet 6ca4fd9cb957431b81257eb461d0b408 (DEFAULT_TABLE table=test-table [id=6149f5fdfb6e42cfbdf387998c06602b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:00:19.755378 8395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ca4fd9cb957431b81257eb461d0b408. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:00:19.755714 8490 tablet_bootstrap.cc:654] T 6ca4fd9cb957431b81257eb461d0b408 P 402c512a26964bd5bc24a332eb2a3860: Neither blocks nor log segments found. Creating new log.
I20260929 21:00:19.753997 8489 tablet_bootstrap.cc:492] T 6ca4fd9cb957431b81257eb461d0b408 P 20f350f99da445e7b1bf54099a7f7e9a: Bootstrap starting.
I20260929 21:00:19.759253 8489 tablet_bootstrap.cc:654] T 6ca4fd9cb957431b81257eb461d0b408 P 20f350f99da445e7b1bf54099a7f7e9a: Neither blocks nor log segments found. Creating new log.
I20260929 21:00:19.760823 8490 tablet_bootstrap.cc:492] T 6ca4fd9cb957431b81257eb461d0b408 P 402c512a26964bd5bc24a332eb2a3860: No bootstrap required, opened a new log
I20260929 21:00:19.760818 8491 tablet_bootstrap.cc:492] T 6ca4fd9cb957431b81257eb461d0b408 P 4407b5bc980b46f88295bed4d965b8a0: Bootstrap starting.
I20260929 21:00:19.761013 8490 ts_tablet_manager.cc:1397] T 6ca4fd9cb957431b81257eb461d0b408 P 402c512a26964bd5bc24a332eb2a3860: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.002s
I20260929 21:00:19.761761 8490 raft_consensus.cc:348] T 6ca4fd9cb957431b81257eb461d0b408 P 402c512a26964bd5bc24a332eb2a3860 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } } peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } }
I20260929 21:00:19.762027 8490 raft_consensus.cc:374] T 6ca4fd9cb957431b81257eb461d0b408 P 402c512a26964bd5bc24a332eb2a3860 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:00:19.762115 8490 raft_consensus.cc:729] T 6ca4fd9cb957431b81257eb461d0b408 P 402c512a26964bd5bc24a332eb2a3860 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 402c512a26964bd5bc24a332eb2a3860, State: Initialized, Role: FOLLOWER
I20260929 21:00:19.762377 8491 tablet_bootstrap.cc:654] T 6ca4fd9cb957431b81257eb461d0b408 P 4407b5bc980b46f88295bed4d965b8a0: Neither blocks nor log segments found. Creating new log.
I20260929 21:00:19.762398 8490 consensus_queue.cc:260] T 6ca4fd9cb957431b81257eb461d0b408 P 402c512a26964bd5bc24a332eb2a3860 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } } peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } }
I20260929 21:00:19.762913 8490 ts_tablet_manager.cc:1428] T 6ca4fd9cb957431b81257eb461d0b408 P 402c512a26964bd5bc24a332eb2a3860: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 21:00:19.764246 7964 catalog_manager.cc:4841] TS 402c512a26964bd5bc24a332eb2a3860 (127.0.97.2:43105): tablet b3e742f6ba3142f89cbfa57e43324f49 (table test-table [id=9ec8f98795ee4f28b5f5a06f6de52e49]) successfully deleted
I20260929 21:00:19.763192 8514 log.cc:1196] T b3e742f6ba3142f89cbfa57e43324f49 P 20f350f99da445e7b1bf54099a7f7e9a: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-0/wal/wals/b3e742f6ba3142f89cbfa57e43324f49
I20260929 21:00:19.764891 8514 ts_tablet_manager.cc:1939] T b3e742f6ba3142f89cbfa57e43324f49 P 20f350f99da445e7b1bf54099a7f7e9a: Deleting consensus metadata
I20260929 21:00:19.765913 7966 catalog_manager.cc:4841] TS 20f350f99da445e7b1bf54099a7f7e9a (127.0.97.1:35627): tablet b3e742f6ba3142f89cbfa57e43324f49 (table test-table [id=9ec8f98795ee4f28b5f5a06f6de52e49]) successfully deleted
I20260929 21:00:19.767539 8489 tablet_bootstrap.cc:492] T 6ca4fd9cb957431b81257eb461d0b408 P 20f350f99da445e7b1bf54099a7f7e9a: No bootstrap required, opened a new log
I20260929 21:00:19.767756 8489 ts_tablet_manager.cc:1397] T 6ca4fd9cb957431b81257eb461d0b408 P 20f350f99da445e7b1bf54099a7f7e9a: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.001s
I20260929 21:00:19.768600 8489 raft_consensus.cc:348] T 6ca4fd9cb957431b81257eb461d0b408 P 20f350f99da445e7b1bf54099a7f7e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } } peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } }
I20260929 21:00:19.768986 8489 raft_consensus.cc:374] T 6ca4fd9cb957431b81257eb461d0b408 P 20f350f99da445e7b1bf54099a7f7e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:00:19.769112 8489 raft_consensus.cc:729] T 6ca4fd9cb957431b81257eb461d0b408 P 20f350f99da445e7b1bf54099a7f7e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20f350f99da445e7b1bf54099a7f7e9a, State: Initialized, Role: FOLLOWER
I20260929 21:00:19.769392 8489 consensus_queue.cc:260] T 6ca4fd9cb957431b81257eb461d0b408 P 20f350f99da445e7b1bf54099a7f7e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } } peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } }
I20260929 21:00:19.770004 8489 ts_tablet_manager.cc:1428] T 6ca4fd9cb957431b81257eb461d0b408 P 20f350f99da445e7b1bf54099a7f7e9a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 21:00:19.770826 8491 tablet_bootstrap.cc:492] T 6ca4fd9cb957431b81257eb461d0b408 P 4407b5bc980b46f88295bed4d965b8a0: No bootstrap required, opened a new log
I20260929 21:00:19.770993 8491 ts_tablet_manager.cc:1397] T 6ca4fd9cb957431b81257eb461d0b408 P 4407b5bc980b46f88295bed4d965b8a0: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260929 21:00:19.771742 8491 raft_consensus.cc:348] T 6ca4fd9cb957431b81257eb461d0b408 P 4407b5bc980b46f88295bed4d965b8a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } } peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } }
I20260929 21:00:19.771976 8491 raft_consensus.cc:374] T 6ca4fd9cb957431b81257eb461d0b408 P 4407b5bc980b46f88295bed4d965b8a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:00:19.772059 8491 raft_consensus.cc:729] T 6ca4fd9cb957431b81257eb461d0b408 P 4407b5bc980b46f88295bed4d965b8a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4407b5bc980b46f88295bed4d965b8a0, State: Initialized, Role: FOLLOWER
I20260929 21:00:19.772326 8491 consensus_queue.cc:260] T 6ca4fd9cb957431b81257eb461d0b408 P 4407b5bc980b46f88295bed4d965b8a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } } peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } }
I20260929 21:00:19.772946 8491 ts_tablet_manager.cc:1428] T 6ca4fd9cb957431b81257eb461d0b408 P 4407b5bc980b46f88295bed4d965b8a0: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260929 21:00:19.887506 8497 raft_consensus.cc:482] T 6ca4fd9cb957431b81257eb461d0b408 P 402c512a26964bd5bc24a332eb2a3860 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 21:00:19.887693 8497 raft_consensus.cc:504] T 6ca4fd9cb957431b81257eb461d0b408 P 402c512a26964bd5bc24a332eb2a3860 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } } peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } }
I20260929 21:00:19.889204 8497 leader_election.cc:288] T 6ca4fd9cb957431b81257eb461d0b408 P 402c512a26964bd5bc24a332eb2a3860 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4407b5bc980b46f88295bed4d965b8a0 (127.0.97.3:35253), 20f350f99da445e7b1bf54099a7f7e9a (127.0.97.1:35627)
I20260929 21:00:19.897399 8570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:19.890177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35627 (local address 127.0.97.2:41309)
0929 21:00:19.890834 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:19.890871 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:19.890979 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:19.891529 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:19.891533 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:19.891554 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:00:19.891851 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:19.891858 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.893302 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.893308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:19.894853 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:19.894868 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.896488 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.896495 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:19.896671 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:19.896708 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 21:00:19.896825 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 21:00:19.897001 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":335,"threads_started":1}
I20260929 21:00:19.897714 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:19.889593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37475 (local address 127.0.97.3:35253)
0929 21:00:19.889792 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:19.889801 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:19.890095 (+ 294us) server_negotiation.cc:407] Connection header received
0929 21:00:19.890864 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:19.890871 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:19.890955 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:19.891049 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:00:19.891514 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.892708 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.895230 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.896461 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.896571 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:19.896680 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:19.897277 (+ 597us) server_negotiation.cc:299] Negotiation successful
0929 21:00:19.897421 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 21:00:19.898066 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:19.890365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41309 (local address 127.0.97.1:35627)
0929 21:00:19.890705 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:19.890711 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:19.890934 (+ 223us) server_negotiation.cc:407] Connection header received
0929 21:00:19.891183 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:19.891190 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:19.891261 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:19.891332 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:00:19.892043 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.893170 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.895004 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.896314 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.896447 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:19.896567 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:19.896873 (+ 306us) server_negotiation.cc:299] Negotiation successful
0929 21:00:19.897407 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207}
I20260929 21:00:19.898409 8569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:19.889518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35253 (local address 127.0.97.2:37475)
0929 21:00:19.889990 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:19.890006 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:19.890174 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:19.891090 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:19.891095 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:19.891123 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:00:19.891355 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:19.891390 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.892928 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.892933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:19.895062 (+ 2129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:19.895078 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.896858 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.896864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:19.897073 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:19.897118 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 21:00:19.897898 (+ 780us) client_negotiation.cc:240] Negotiation successful
0929 21:00:19.898035 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":4,"thread_start_us":275,"threads_started":1}
I20260929 21:00:19.898676 8141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ca4fd9cb957431b81257eb461d0b408" candidate_uuid: "402c512a26964bd5bc24a332eb2a3860" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20f350f99da445e7b1bf54099a7f7e9a" is_pre_election: true
I20260929 21:00:19.898957 8420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ca4fd9cb957431b81257eb461d0b408" candidate_uuid: "402c512a26964bd5bc24a332eb2a3860" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4407b5bc980b46f88295bed4d965b8a0" is_pre_election: true
I20260929 21:00:19.898975 8141 raft_consensus.cc:2454] T 6ca4fd9cb957431b81257eb461d0b408 P 20f350f99da445e7b1bf54099a7f7e9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 402c512a26964bd5bc24a332eb2a3860 in term 0.
I20260929 21:00:19.899137 8420 raft_consensus.cc:2454] T 6ca4fd9cb957431b81257eb461d0b408 P 4407b5bc980b46f88295bed4d965b8a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 402c512a26964bd5bc24a332eb2a3860 in term 0.
I20260929 21:00:19.899525 8209 leader_election.cc:302] T 6ca4fd9cb957431b81257eb461d0b408 P 402c512a26964bd5bc24a332eb2a3860 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20f350f99da445e7b1bf54099a7f7e9a, 402c512a26964bd5bc24a332eb2a3860; no voters:
I20260929 21:00:19.899885 8497 raft_consensus.cc:2781] T 6ca4fd9cb957431b81257eb461d0b408 P 402c512a26964bd5bc24a332eb2a3860 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 21:00:19.900014 8497 raft_consensus.cc:482] T 6ca4fd9cb957431b81257eb461d0b408 P 402c512a26964bd5bc24a332eb2a3860 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 21:00:19.900213 8497 raft_consensus.cc:3037] T 6ca4fd9cb957431b81257eb461d0b408 P 402c512a26964bd5bc24a332eb2a3860 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:00:19.902485 8497 raft_consensus.cc:504] T 6ca4fd9cb957431b81257eb461d0b408 P 402c512a26964bd5bc24a332eb2a3860 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } } peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } }
I20260929 21:00:19.903048 8497 leader_election.cc:288] T 6ca4fd9cb957431b81257eb461d0b408 P 402c512a26964bd5bc24a332eb2a3860 [CANDIDATE]: Term 1 election: Requested vote from peers 4407b5bc980b46f88295bed4d965b8a0 (127.0.97.3:35253), 20f350f99da445e7b1bf54099a7f7e9a (127.0.97.1:35627)
I20260929 21:00:19.903626 8141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ca4fd9cb957431b81257eb461d0b408" candidate_uuid: "402c512a26964bd5bc24a332eb2a3860" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20f350f99da445e7b1bf54099a7f7e9a"
I20260929 21:00:19.903611 8420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ca4fd9cb957431b81257eb461d0b408" candidate_uuid: "402c512a26964bd5bc24a332eb2a3860" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4407b5bc980b46f88295bed4d965b8a0"
I20260929 21:00:19.903831 8141 raft_consensus.cc:3037] T 6ca4fd9cb957431b81257eb461d0b408 P 20f350f99da445e7b1bf54099a7f7e9a [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:00:19.903842 8420 raft_consensus.cc:3037] T 6ca4fd9cb957431b81257eb461d0b408 P 4407b5bc980b46f88295bed4d965b8a0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:00:19.906056 8141 raft_consensus.cc:2454] T 6ca4fd9cb957431b81257eb461d0b408 P 20f350f99da445e7b1bf54099a7f7e9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 402c512a26964bd5bc24a332eb2a3860 in term 1.
I20260929 21:00:19.906456 8209 leader_election.cc:302] T 6ca4fd9cb957431b81257eb461d0b408 P 402c512a26964bd5bc24a332eb2a3860 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20f350f99da445e7b1bf54099a7f7e9a, 402c512a26964bd5bc24a332eb2a3860; no voters:
I20260929 21:00:19.906780 8497 raft_consensus.cc:2781] T 6ca4fd9cb957431b81257eb461d0b408 P 402c512a26964bd5bc24a332eb2a3860 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:00:19.906833 8420 raft_consensus.cc:2454] T 6ca4fd9cb957431b81257eb461d0b408 P 4407b5bc980b46f88295bed4d965b8a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 402c512a26964bd5bc24a332eb2a3860 in term 1.
I20260929 21:00:19.907688 8497 raft_consensus.cc:686] T 6ca4fd9cb957431b81257eb461d0b408 P 402c512a26964bd5bc24a332eb2a3860 [term 1 LEADER]: Becoming Leader. State: Replica: 402c512a26964bd5bc24a332eb2a3860, State: Running, Role: LEADER
I20260929 21:00:19.908293 8497 consensus_queue.cc:237] T 6ca4fd9cb957431b81257eb461d0b408 P 402c512a26964bd5bc24a332eb2a3860 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } } peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } }
I20260929 21:00:19.912297 7978 catalog_manager.cc:5495] T 6ca4fd9cb957431b81257eb461d0b408 P 402c512a26964bd5bc24a332eb2a3860 reported cstate change: term changed from 0 to 1, leader changed from <none> to 402c512a26964bd5bc24a332eb2a3860 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "402c512a26964bd5bc24a332eb2a3860" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "402c512a26964bd5bc24a332eb2a3860" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43105 } health_report { overall_health: HEALTHY } } }
I20260929 21:00:19.971668 8573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:19.961675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50520 (local address 127.0.97.2:43105)
0929 21:00:19.965466 (+ 3791us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:19.965472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:19.965494 (+ 22us) server_negotiation.cc:407] Connection header received
0929 21:00:19.965593 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:19.965599 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:19.965722 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:19.965902 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:00:19.966313 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.967489 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.969008 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.969678 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.969812 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:19.970226 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:00:19.970973 (+ 747us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:00:19.971114 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:19.971212 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 21:00:19.971299 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3661,"thread_start_us":3504,"threads_started":1}
I20260929 21:00:19.987465 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:19.981929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33901 (local address 127.0.97.1:35627)
0929 21:00:19.982113 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:19.982119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:19.982709 (+ 590us) server_negotiation.cc:407] Connection header received
0929 21:00:19.982918 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:19.982923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:19.983011 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:19.983112 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:00:19.983568 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.984338 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.985712 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.986142 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.986290 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:19.986699 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:00:19.986826 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:00:19.986925 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:19.987061 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 21:00:19.987132 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 21:00:19.987540 8576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:19.981929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35627 (local address 127.0.97.2:33901)
0929 21:00:19.982613 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:19.982628 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:19.982777 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:19.983118 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:19.983123 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:19.983134 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 21:00:19.983429 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:19.983437 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.984469 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.984475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:19.985555 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:19.985566 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:19.986331 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:19.986341 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:19.986477 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:19.986523 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 21:00:19.986925 (+ 402us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 21:00:19.986945 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 21:00:19.987041 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 21:00:19.987161 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":379,"threads_started":1}
I20260929 21:00:19.996196 8141 raft_consensus.cc:1261] T 6ca4fd9cb957431b81257eb461d0b408 P 20f350f99da445e7b1bf54099a7f7e9a [term 1 FOLLOWER]: Refusing update from remote peer 402c512a26964bd5bc24a332eb2a3860: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:00:19.996197 8420 raft_consensus.cc:1261] T 6ca4fd9cb957431b81257eb461d0b408 P 4407b5bc980b46f88295bed4d965b8a0 [term 1 FOLLOWER]: Refusing update from remote peer 402c512a26964bd5bc24a332eb2a3860: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:00:19.996852 8571 consensus_queue.cc:1035] T 6ca4fd9cb957431b81257eb461d0b408 P 402c512a26964bd5bc24a332eb2a3860 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4407b5bc980b46f88295bed4d965b8a0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:00:19.997264 8497 consensus_queue.cc:1035] T 6ca4fd9cb957431b81257eb461d0b408 P 402c512a26964bd5bc24a332eb2a3860 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20f350f99da445e7b1bf54099a7f7e9a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:00:20.052292 8573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:20.035361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40457 (local address 127.0.97.2:43105)
0929 21:00:20.035526 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:20.035534 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:20.045871 (+ 10337us) server_negotiation.cc:407] Connection header received
0929 21:00:20.045967 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:20.045972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:20.046045 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:20.046135 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:00:20.046697 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:20.047834 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:20.050316 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:20.050990 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:20.051108 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:20.051559 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:00:20.051677 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:00:20.051759 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:20.051915 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 21:00:20.051991 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 21:00:20.052500 8580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:20.035338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43105 (local address 127.0.97.1:40457)
0929 21:00:20.045625 (+ 10287us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:20.045662 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:20.045791 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:20.046165 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:20.046170 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:20.046183 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 21:00:20.046527 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:20.046538 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:20.048137 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:20.048142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:20.050164 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:20.050185 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:20.051149 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:20.051155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:20.051349 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:20.051398 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 21:00:20.051783 (+ 385us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 21:00:20.051795 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 21:00:20.051883 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 21:00:20.052027 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10132,"thread_start_us":9980,"threads_started":1}
I20260929 21:00:20.098897 7978 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 21:00:20.099148 7978 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 21:00:20.102399 7978 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c5be66ba68f48a2a21988e7a837996c: Sending DeleteTablet for 3 replicas of tablet 6ca4fd9cb957431b81257eb461d0b408
I20260929 21:00:20.103300 8395 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ca4fd9cb957431b81257eb461d0b408 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:00:20 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41072
I20260929 21:00:20.103580 8513 tablet_replica.cc:330] T 6ca4fd9cb957431b81257eb461d0b408 P 4407b5bc980b46f88295bed4d965b8a0: stopping tablet replica
I20260929 21:00:20.103590 8256 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ca4fd9cb957431b81257eb461d0b408 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:00:20 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50516
I20260929 21:00:20.103773 8513 raft_consensus.cc:2229] T 6ca4fd9cb957431b81257eb461d0b408 P 4407b5bc980b46f88295bed4d965b8a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:00:20.103889 8512 tablet_replica.cc:330] T 6ca4fd9cb957431b81257eb461d0b408 P 402c512a26964bd5bc24a332eb2a3860: stopping tablet replica
I20260929 21:00:20.103952 8513 raft_consensus.cc:2258] T 6ca4fd9cb957431b81257eb461d0b408 P 4407b5bc980b46f88295bed4d965b8a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:00:20.104045 8512 raft_consensus.cc:2229] T 6ca4fd9cb957431b81257eb461d0b408 P 402c512a26964bd5bc24a332eb2a3860 [term 1 LEADER]: Raft consensus shutting down.
I20260929 21:00:20.104529 8512 raft_consensus.cc:2258] T 6ca4fd9cb957431b81257eb461d0b408 P 402c512a26964bd5bc24a332eb2a3860 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:00:20.104665 8117 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ca4fd9cb957431b81257eb461d0b408 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:00:20 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44256
I20260929 21:00:20.105170 8514 tablet_replica.cc:330] T 6ca4fd9cb957431b81257eb461d0b408 P 20f350f99da445e7b1bf54099a7f7e9a: stopping tablet replica
I20260929 21:00:20.105227 8513 ts_tablet_manager.cc:1905] T 6ca4fd9cb957431b81257eb461d0b408 P 4407b5bc980b46f88295bed4d965b8a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:00:20.105367 8514 raft_consensus.cc:2229] T 6ca4fd9cb957431b81257eb461d0b408 P 20f350f99da445e7b1bf54099a7f7e9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:00:20.105547 8514 raft_consensus.cc:2258] T 6ca4fd9cb957431b81257eb461d0b408 P 20f350f99da445e7b1bf54099a7f7e9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:00:20.106674 8514 ts_tablet_manager.cc:1905] T 6ca4fd9cb957431b81257eb461d0b408 P 20f350f99da445e7b1bf54099a7f7e9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:00:20.107399 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 8051
I20260929 21:00:20.109932 8512 ts_tablet_manager.cc:1905] T 6ca4fd9cb957431b81257eb461d0b408 P 402c512a26964bd5bc24a332eb2a3860: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:00:20.113601 8514 ts_tablet_manager.cc:1918] T 6ca4fd9cb957431b81257eb461d0b408 P 20f350f99da445e7b1bf54099a7f7e9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 21:00:20.113741 8514 log.cc:1196] T 6ca4fd9cb957431b81257eb461d0b408 P 20f350f99da445e7b1bf54099a7f7e9a: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-0/wal/wals/6ca4fd9cb957431b81257eb461d0b408
I20260929 21:00:20.114147 8514 ts_tablet_manager.cc:1939] T 6ca4fd9cb957431b81257eb461d0b408 P 20f350f99da445e7b1bf54099a7f7e9a: Deleting consensus metadata
I20260929 21:00:20.114674 8513 ts_tablet_manager.cc:1918] T 6ca4fd9cb957431b81257eb461d0b408 P 4407b5bc980b46f88295bed4d965b8a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 21:00:20.114843 8513 log.cc:1196] T 6ca4fd9cb957431b81257eb461d0b408 P 4407b5bc980b46f88295bed4d965b8a0: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-2/wal/wals/6ca4fd9cb957431b81257eb461d0b408
I20260929 21:00:20.115303 8513 ts_tablet_manager.cc:1939] T 6ca4fd9cb957431b81257eb461d0b408 P 4407b5bc980b46f88295bed4d965b8a0: Deleting consensus metadata
I20260929 21:00:20.115595 7966 catalog_manager.cc:4841] TS 20f350f99da445e7b1bf54099a7f7e9a (127.0.97.1:35627): tablet 6ca4fd9cb957431b81257eb461d0b408 (table test-table [id=6149f5fdfb6e42cfbdf387998c06602b]) successfully deleted
I20260929 21:00:20.116484 7966 catalog_manager.cc:4841] TS 4407b5bc980b46f88295bed4d965b8a0 (127.0.97.3:35253): tablet 6ca4fd9cb957431b81257eb461d0b408 (table test-table [id=6149f5fdfb6e42cfbdf387998c06602b]) successfully deleted
I20260929 21:00:20.118058 8512 ts_tablet_manager.cc:1918] T 6ca4fd9cb957431b81257eb461d0b408 P 402c512a26964bd5bc24a332eb2a3860: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 21:00:20.118229 8512 log.cc:1196] T 6ca4fd9cb957431b81257eb461d0b408 P 402c512a26964bd5bc24a332eb2a3860: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790715556572573-388-0/minicluster-data/ts-1/wal/wals/6ca4fd9cb957431b81257eb461d0b408
I20260929 21:00:20.118649 8512 ts_tablet_manager.cc:1939] T 6ca4fd9cb957431b81257eb461d0b408 P 402c512a26964bd5bc24a332eb2a3860: Deleting consensus metadata
I20260929 21:00:20.119731 7964 catalog_manager.cc:4841] TS 402c512a26964bd5bc24a332eb2a3860 (127.0.97.2:43105): tablet 6ca4fd9cb957431b81257eb461d0b408 (table test-table [id=6149f5fdfb6e42cfbdf387998c06602b]) successfully deleted
Sep 29 21:00:20 dist-test-slave-1dms krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260929 21:00:20.138780 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:20.108745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44276 (local address 127.0.97.1:35627)
0929 21:00:20.109180 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:20.109186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:20.120433 (+ 11247us) server_negotiation.cc:407] Connection header received
0929 21:00:20.120646 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:20.120654 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:20.120752 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:20.120869 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:20.122595 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:20.123733 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:20.125537 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:20.126358 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:20.126491 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:20.131479 (+ 4988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:20.131497 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:20.131519 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:20.131565 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:20.135647 (+ 4082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:20.136474 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:20.136489 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:20.136492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:20.136664 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:20.137105 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:20.137110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:20.137113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:20.137584 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:20.137985 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:20.138184 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 21:00:20.138408 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269}
I20260929 21:00:20.139640 8182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:00:20.294078 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 8051
I20260929 21:00:20.318636 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 8191
Sep 29 21:00:20 dist-test-slave-1dms krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260929 21:00:20.335770 8573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:20.319489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50532 (local address 127.0.97.2:43105)
0929 21:00:20.319698 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:20.319705 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:20.319787 (+ 82us) server_negotiation.cc:407] Connection header received
0929 21:00:20.320018 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:20.320022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:20.320087 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:20.320232 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:20.321504 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:20.322281 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:20.323501 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:20.323978 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:20.324113 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:20.328449 (+ 4336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:20.328462 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:20.328480 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:20.328522 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:20.332710 (+ 4188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:20.333490 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:20.333500 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:20.333504 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:20.333664 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:20.334050 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:20.334057 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:20.334062 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:20.334528 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:20.334859 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:20.335117 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 21:00:20.335357 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 21:00:20.336509 8321 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:00:20.465018 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 8191
I20260929 21:00:20.486039 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 8330
Sep 29 21:00:20 dist-test-slave-1dms krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260929 21:00:20.500643 8587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:20.486763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41090 (local address 127.0.97.3:35253)
0929 21:00:20.487249 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:20.487257 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:20.487271 (+ 14us) server_negotiation.cc:407] Connection header received
0929 21:00:20.487322 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:20.487326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:20.487380 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:20.487508 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:20.488758 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:20.489501 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:20.490705 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:20.491190 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:20.491314 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:20.495547 (+ 4233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:20.495561 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:20.495577 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:20.495615 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:20.498313 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:20.498907 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:20.498914 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:20.498916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:20.499054 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:20.499346 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:20.499349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:20.499351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:20.499675 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:20.499893 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:20.500107 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 21:00:20.500260 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":234,"threads_started":1}
I20260929 21:00:20.501734 8460 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:00:20.626494 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 8330
I20260929 21:00:20.648521 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 7948
I20260929 21:00:20.649672 8020 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:00:20.761847 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 7948
2026-09-29T21:00:20Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5565 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 21:00:20 dist-test-slave-1dms krb5kdc[8592](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 21:00:20 dist-test-slave-1dms krb5kdc[8592](info): set up 2 sockets
Sep 29 21:00:20 dist-test-slave-1dms krb5kdc[8592](info): commencing operation
krb5kdc: starting...
W20260929 21:00:21.857834 388 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.003s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 21:00:21 dist-test-slave-1dms krb5kdc[8592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T21:00:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:00:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 21:00:22.080638 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44549
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:00:22.392176 8605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:22.392501 8605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:22.392678 8605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:22.399750 8605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:00:22.399888 8605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:22.399994 8605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:22.400110 8605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:00:22.400211 8605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:00:22.410056 8605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:00:22.411322 8605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:22.412614 8605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:22.423985 8611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:22.423956 8613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:22.424320 8610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:22.424881 8605 server_base.cc:1034] running on GCE node
I20260929 21:00:22.425638 8605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:22.427007 8605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:22.428215 8605 hybrid_clock.cc:648] HybridClock initialized: now 1790715622428162 us; error 37 us; skew 500 ppm
I20260929 21:00:22.428576 8605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T21:00:22Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1733 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 21:00:22 dist-test-slave-1dms krb5kdc[8619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 21:00:22 dist-test-slave-1dms krb5kdc[8619](info): set up 2 sockets
Sep 29 21:00:22 dist-test-slave-1dms krb5kdc[8619](info): commencing operation
krb5kdc: starting...
W20260929 21:00:23.609874 388 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 21:00:23 dist-test-slave-1dms krb5kdc[8619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T21:00:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:00:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 21:00:23.814709 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39497
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39497
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:00:24.129554 8632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:24.129844 8632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:24.130045 8632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:24.137375 8632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:00:24.137494 8632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:24.137630 8632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:24.137729 8632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:00:24.137804 8632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:00:24.147778 8632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:00:24.149183 8632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:24.150488 8632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:24.161743 8637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:24.161698 8638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:24.161844 8640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:24.162794 8632 server_base.cc:1034] running on GCE node
I20260929 21:00:24.163678 8632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:24.165042 8632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:24.166236 8632 hybrid_clock.cc:648] HybridClock initialized: now 1790715624166200 us; error 29 us; skew 500 ppm
I20260929 21:00:24.166563 8632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:00:24 dist-test-slave-1dms krb5kdc[8619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:00:24.177397 8632 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 21:00:24.178982 8632 webserver.cc:460] Webserver started at http://127.0.97.62:43615/ using document root <none> and password file <none>
I20260929 21:00:24.179688 8632 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:24.179834 8632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:24.180155 8632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:00:24.183159 8632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "c75b9414215e432c84a48581b0f3bad3"
format_stamp: "Formatted at 2026-09-29 21:00:24 on dist-test-slave-1dms"
server_key: "32aa2ce6f64d759e28f0e9be3d5e0829"
server_key_iv: "eff210887a351b2bd8f9d0427f24a24f"
server_key_version: "encryptionkey@0"
I20260929 21:00:24.183944 8632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "c75b9414215e432c84a48581b0f3bad3"
format_stamp: "Formatted at 2026-09-29 21:00:24 on dist-test-slave-1dms"
server_key: "32aa2ce6f64d759e28f0e9be3d5e0829"
server_key_iv: "eff210887a351b2bd8f9d0427f24a24f"
server_key_version: "encryptionkey@0"
I20260929 21:00:24.188926 8632 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 21:00:24.192163 8647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:24.193454 8632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 21:00:24.193632 8632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "c75b9414215e432c84a48581b0f3bad3"
format_stamp: "Formatted at 2026-09-29 21:00:24 on dist-test-slave-1dms"
server_key: "32aa2ce6f64d759e28f0e9be3d5e0829"
server_key_iv: "eff210887a351b2bd8f9d0427f24a24f"
server_key_version: "encryptionkey@0"
I20260929 21:00:24.193811 8632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:24.212431 8632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:24.216214 8632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:00:24.216486 8632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:24.234484 8632 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39497
I20260929 21:00:24.234516 8698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39497 every 8 connection(s)
I20260929 21:00:24.235809 8632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:00:24.239557 8699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:00:24.244201 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 8632
I20260929 21:00:24.244417 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:00:24.245247 388 external_mini_cluster.cc:1428] Setting key 188006ccdc675fb402dac39417742203
I20260929 21:00:24.250765 8699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3: Bootstrap starting.
I20260929 21:00:24.253787 8699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3: Neither blocks nor log segments found. Creating new log.
I20260929 21:00:24.254745 8699 log.cc:824] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3: Log is configured to *not* fsync() on all Append() calls
I20260929 21:00:24.258073 8699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3: No bootstrap required, opened a new log
Sep 29 21:00:24 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:00:24.268715 8699 raft_consensus.cc:348] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c75b9414215e432c84a48581b0f3bad3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39497 } }
I20260929 21:00:24.269193 8699 raft_consensus.cc:374] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:00:24.269215 8702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:24.247282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38992 (local address 127.0.97.62:39497)
0929 21:00:24.248045 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:24.248061 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:24.248102 (+ 41us) server_negotiation.cc:407] Connection header received
0929 21:00:24.249889 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:24.249914 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:24.250293 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:24.250688 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:24.252121 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:24.253471 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:24.254774 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:24.255844 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:24.256044 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:24.260604 (+ 4560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:24.260616 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:24.260642 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:24.260702 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:24.264691 (+ 3989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:24.265408 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:24.265417 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:24.265427 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:24.265664 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:24.265988 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:24.265993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:24.265997 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:24.266763 (+ 766us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:24.267075 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:24.267361 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 21:00:24.267821 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":334,"threads_started":1}
I20260929 21:00:24.269339 8699 raft_consensus.cc:729] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c75b9414215e432c84a48581b0f3bad3, State: Initialized, Role: FOLLOWER
I20260929 21:00:24.269963 8699 consensus_queue.cc:260] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c75b9414215e432c84a48581b0f3bad3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39497 } }
I20260929 21:00:24.270236 8699 raft_consensus.cc:388] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:00:24.270368 8699 raft_consensus.cc:482] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:00:24.270545 8699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:00:24.273046 8699 raft_consensus.cc:504] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c75b9414215e432c84a48581b0f3bad3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39497 } }
I20260929 21:00:24.273469 8699 leader_election.cc:302] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c75b9414215e432c84a48581b0f3bad3; no voters:
I20260929 21:00:24.273993 8699 leader_election.cc:288] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 21:00:24.274143 8704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:00:24.274780 8704 raft_consensus.cc:686] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3 [term 1 LEADER]: Becoming Leader. State: Replica: c75b9414215e432c84a48581b0f3bad3, State: Running, Role: LEADER
I20260929 21:00:24.275318 8704 consensus_queue.cc:237] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c75b9414215e432c84a48581b0f3bad3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39497 } }
I20260929 21:00:24.276028 8699 sys_catalog.cc:564] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:00:24.278613 8705 sys_catalog.cc:455] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c75b9414215e432c84a48581b0f3bad3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c75b9414215e432c84a48581b0f3bad3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39497 } } }
I20260929 21:00:24.279059 8705 sys_catalog.cc:458] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3 [sys.catalog]: This master's current role is: LEADER
I20260929 21:00:24.280508 8706 sys_catalog.cc:455] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c75b9414215e432c84a48581b0f3bad3. Latest consensus state: current_term: 1 leader_uuid: "c75b9414215e432c84a48581b0f3bad3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c75b9414215e432c84a48581b0f3bad3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39497 } } }
I20260929 21:00:24.280767 8706 sys_catalog.cc:458] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3 [sys.catalog]: This master's current role is: LEADER
I20260929 21:00:24.284821 8713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:00:24.289934 8713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:00:24.298736 8713 catalog_manager.cc:1347] Generated new cluster ID: 589fc8135e094ae389b68848ed86b316
I20260929 21:00:24.298851 8713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:00:24.333632 8713 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:00:24.334982 8713 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:00:24.345676 8713 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3: Generated new TSK 0
I20260929 21:00:24.346524 8713 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 21:00:24.406440 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39497
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:00:24.738698 8725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:24.738962 8725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:24.739053 8725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:24.746021 8725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:24.746122 8725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:24.746230 8725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 21:00:24.755550 8725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:00:24.756690 8725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:24.758227 8725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:24.770638 8730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:24.770697 8731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:24.770668 8733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:24.771806 8725 server_base.cc:1034] running on GCE node
I20260929 21:00:24.772341 8725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:24.773283 8725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:24.774509 8725 hybrid_clock.cc:648] HybridClock initialized: now 1790715624774471 us; error 50 us; skew 500 ppm
I20260929 21:00:24.774924 8725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:00:24 dist-test-slave-1dms krb5kdc[8619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:00:24.778281 8725 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 21:00:24.779894 8725 webserver.cc:460] Webserver started at http://127.0.97.1:42483/ using document root <none> and password file <none>
I20260929 21:00:24.780610 8725 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:24.780754 8725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:24.781112 8725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:00:24.783895 8725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-0/data/instance:
uuid: "24fa043915444d9199aaac5ddae48217"
format_stamp: "Formatted at 2026-09-29 21:00:24 on dist-test-slave-1dms"
server_key: "e2221f930d1eeec6473de3708a7e3dce"
server_key_iv: "b3c98884de0c8c583a742c32584f7960"
server_key_version: "encryptionkey@0"
I20260929 21:00:24.784650 8725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-0/wal/instance:
uuid: "24fa043915444d9199aaac5ddae48217"
format_stamp: "Formatted at 2026-09-29 21:00:24 on dist-test-slave-1dms"
server_key: "e2221f930d1eeec6473de3708a7e3dce"
server_key_iv: "b3c98884de0c8c583a742c32584f7960"
server_key_version: "encryptionkey@0"
I20260929 21:00:24.789825 8725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 21:00:24.793159 8740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:24.794379 8725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 21:00:24.794541 8725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-0/wal
uuid: "24fa043915444d9199aaac5ddae48217"
format_stamp: "Formatted at 2026-09-29 21:00:24 on dist-test-slave-1dms"
server_key: "e2221f930d1eeec6473de3708a7e3dce"
server_key_iv: "b3c98884de0c8c583a742c32584f7960"
server_key_version: "encryptionkey@0"
I20260929 21:00:24.794745 8725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:24.814795 8725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:24.818701 8725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:00:24.818994 8725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:24.819909 8725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:00:24.821233 8725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:00:24.821342 8725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:24.821494 8725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:00:24.821565 8725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:24.853983 8725 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43645
I20260929 21:00:24.854039 8852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43645 every 8 connection(s)
I20260929 21:00:24.855262 8725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
I20260929 21:00:24.859959 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 8725
I20260929 21:00:24.860162 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-0/wal/instance
I20260929 21:00:24.860777 388 external_mini_cluster.cc:1428] Setting key c80835b92734c4ec6d17c95aa05417e4
Sep 29 21:00:24 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:00:24.878500 8855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:24.857946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46139 (local address 127.0.97.62:39497)
0929 21:00:24.858365 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:24.858370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:24.859358 (+ 988us) server_negotiation.cc:407] Connection header received
0929 21:00:24.861336 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:24.861341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:24.861435 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:24.861554 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:24.863705 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:24.864420 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:24.866975 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:24.867380 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:24.867486 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:24.871586 (+ 4100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:24.871595 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:24.871603 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:24.871647 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:24.875128 (+ 3481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:24.875907 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:24.875916 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:24.875923 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:24.876064 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:24.876483 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:24.876491 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:24.876497 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:24.876895 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:24.877037 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:24.877771 (+ 734us) server_negotiation.cc:299] Negotiation successful
0929 21:00:24.878015 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":217,"threads_started":1}
I20260929 21:00:24.879197 8856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:24.858536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.1:46139)
0929 21:00:24.859204 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:24.859247 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:24.861070 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:24.862071 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:24.862083 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:24.862737 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:24.863518 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:24.863547 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:24.864593 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:24.864601 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:24.866822 (+ 2221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:24.866839 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:24.867611 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:24.867621 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:24.867944 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:24.868677 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:24.868718 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:24.871354 (+ 2636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:24.875270 (+ 3916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:24.875276 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:24.875290 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:24.875770 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:24.876171 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:24.876174 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:24.876177 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:24.876354 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:24.877029 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:24.877037 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:24.877375 (+ 338us) client_negotiation.cc:769] Sending connection context
0929 21:00:24.877608 (+ 233us) client_negotiation.cc:240] Negotiation successful
0929 21:00:24.878012 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":262,"threads_started":1}
I20260929 21:00:24.880951 8853 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39497
I20260929 21:00:24.881258 8853 heartbeater.cc:464] Registering TS with master...
I20260929 21:00:24.881968 8853 heartbeater.cc:511] Master 127.0.97.62:39497 requested a full tablet report, sending...
I20260929 21:00:24.884021 8664 ts_manager.cc:194] Registered new tserver with Master: 24fa043915444d9199aaac5ddae48217 (127.0.97.1:43645)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 21:00:24.885614 8664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46139
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 21:00:24.912377 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39497
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:00:25.235456 8859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:25.235852 8859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:25.236004 8859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:25.247601 8859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:25.247743 8859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:25.247923 8859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 21:00:25.261430 8859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:00:25.262663 8859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:25.263871 8859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:25.276929 8865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:25.276868 8864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:25.276908 8867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:25.277854 8859 server_base.cc:1034] running on GCE node
I20260929 21:00:25.278497 8859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:25.279439 8859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:25.280682 8859 hybrid_clock.cc:648] HybridClock initialized: now 1790715625280594 us; error 65 us; skew 500 ppm
I20260929 21:00:25.281049 8859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:00:25 dist-test-slave-1dms krb5kdc[8619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:00:25.284003 8859 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 21:00:25.285498 8859 webserver.cc:460] Webserver started at http://127.0.97.2:43993/ using document root <none> and password file <none>
I20260929 21:00:25.286375 8859 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:25.286486 8859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:25.286779 8859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:00:25.289687 8859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-1/data/instance:
uuid: "67c1353efcd249daa78705582ab51a13"
format_stamp: "Formatted at 2026-09-29 21:00:25 on dist-test-slave-1dms"
server_key: "3ada611c76ab6535f5dd62511b7ffb4b"
server_key_iv: "0e24adc29d9172fc0507dbf4eb8f0095"
server_key_version: "encryptionkey@0"
I20260929 21:00:25.290410 8859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-1/wal/instance:
uuid: "67c1353efcd249daa78705582ab51a13"
format_stamp: "Formatted at 2026-09-29 21:00:25 on dist-test-slave-1dms"
server_key: "3ada611c76ab6535f5dd62511b7ffb4b"
server_key_iv: "0e24adc29d9172fc0507dbf4eb8f0095"
server_key_version: "encryptionkey@0"
I20260929 21:00:25.295528 8859 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 21:00:25.299180 8874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:25.300565 8859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 21:00:25.300701 8859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-1/wal
uuid: "67c1353efcd249daa78705582ab51a13"
format_stamp: "Formatted at 2026-09-29 21:00:25 on dist-test-slave-1dms"
server_key: "3ada611c76ab6535f5dd62511b7ffb4b"
server_key_iv: "0e24adc29d9172fc0507dbf4eb8f0095"
server_key_version: "encryptionkey@0"
I20260929 21:00:25.300956 8859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:25.315558 8859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:25.319893 8859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:00:25.320176 8859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:25.321125 8859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:00:25.322373 8859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:00:25.322475 8859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:25.322629 8859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:00:25.322705 8859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:25.353346 8859 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44387
I20260929 21:00:25.353394 8986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44387 every 8 connection(s)
I20260929 21:00:25.354568 8859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
I20260929 21:00:25.362247 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 8859
I20260929 21:00:25.362414 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-1/wal/instance
I20260929 21:00:25.362860 388 external_mini_cluster.cc:1428] Setting key 10f04b365c814f1fdff7487b3155d161
Sep 29 21:00:25 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:00:25.375600 8855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:25.357089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35945 (local address 127.0.97.62:39497)
0929 21:00:25.357258 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:25.357262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:25.358455 (+ 1193us) server_negotiation.cc:407] Connection header received
0929 21:00:25.360292 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:25.360298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:25.360384 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:25.360497 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:25.362009 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:25.362808 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:25.364808 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:25.365455 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:25.365591 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:25.369239 (+ 3648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:25.369249 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:25.369256 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:25.369302 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:25.372447 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:25.373385 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:25.373389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:25.373392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:25.373518 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:25.374000 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:25.374003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:25.374005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:25.374250 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:25.374351 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:25.375208 (+ 857us) server_negotiation.cc:299] Negotiation successful
0929 21:00:25.375368 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 21:00:25.377213 8989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:25.357594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.2:35945)
0929 21:00:25.358290 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:25.358332 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:25.360022 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:25.360721 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:25.360729 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:25.361267 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:25.361800 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:25.361816 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:25.362977 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:25.362985 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:25.364653 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:25.364664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:25.365677 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:25.365682 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:25.365875 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:25.366665 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:25.366687 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:25.369022 (+ 2335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:25.372594 (+ 3572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:25.372602 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:25.372620 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:25.373251 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:25.373627 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:25.373635 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:25.373642 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:25.373882 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:25.374365 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:25.374376 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:25.374819 (+ 443us) client_negotiation.cc:769] Sending connection context
0929 21:00:25.375136 (+ 317us) client_negotiation.cc:240] Negotiation successful
0929 21:00:25.375705 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":279,"threads_started":1}
I20260929 21:00:25.378631 8987 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39497
I20260929 21:00:25.378961 8987 heartbeater.cc:464] Registering TS with master...
I20260929 21:00:25.379832 8987 heartbeater.cc:511] Master 127.0.97.62:39497 requested a full tablet report, sending...
I20260929 21:00:25.381403 8664 ts_manager.cc:194] Registered new tserver with Master: 67c1353efcd249daa78705582ab51a13 (127.0.97.2:44387)
I20260929 21:00:25.382187 8664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35945
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 21:00:25.414255 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39497
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:00:25.734387 8992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:25.734694 8992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:25.734858 8992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:25.741871 8992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:25.742012 8992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:25.742187 8992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 21:00:25.751494 8992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:00:25.752717 8992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:25.753985 8992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:25.766764 8998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:25.766908 8997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:25.767212 9000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:25.768096 8992 server_base.cc:1034] running on GCE node
I20260929 21:00:25.768678 8992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:25.769603 8992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:25.770814 8992 hybrid_clock.cc:648] HybridClock initialized: now 1790715625770781 us; error 45 us; skew 500 ppm
I20260929 21:00:25.771126 8992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:00:25 dist-test-slave-1dms krb5kdc[8619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:00:25.774361 8992 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 21:00:25.775699 8992 webserver.cc:460] Webserver started at http://127.0.97.3:35789/ using document root <none> and password file <none>
I20260929 21:00:25.776311 8992 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:25.776433 8992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:25.776736 8992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:00:25.779641 8992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-2/data/instance:
uuid: "83207e7121784978b030c97faeeb997f"
format_stamp: "Formatted at 2026-09-29 21:00:25 on dist-test-slave-1dms"
server_key: "12d61f2a8ed88fa4f7943ce3b1a63ac3"
server_key_iv: "64e369750287b3b7ddc5e5248f861f0e"
server_key_version: "encryptionkey@0"
I20260929 21:00:25.780385 8992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-2/wal/instance:
uuid: "83207e7121784978b030c97faeeb997f"
format_stamp: "Formatted at 2026-09-29 21:00:25 on dist-test-slave-1dms"
server_key: "12d61f2a8ed88fa4f7943ce3b1a63ac3"
server_key_iv: "64e369750287b3b7ddc5e5248f861f0e"
server_key_version: "encryptionkey@0"
I20260929 21:00:25.785166 8992 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 21:00:25.788323 9007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:25.789709 8992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:00:25.789875 8992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-2/wal
uuid: "83207e7121784978b030c97faeeb997f"
format_stamp: "Formatted at 2026-09-29 21:00:25 on dist-test-slave-1dms"
server_key: "12d61f2a8ed88fa4f7943ce3b1a63ac3"
server_key_iv: "64e369750287b3b7ddc5e5248f861f0e"
server_key_version: "encryptionkey@0"
I20260929 21:00:25.790077 8992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:25.813350 8992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:25.817677 8992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:00:25.817976 8992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:25.818944 8992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:00:25.820194 8992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:00:25.820263 8992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:25.820436 8992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:00:25.820487 8992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:25.852417 8992 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42169
I20260929 21:00:25.852474 9119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42169 every 8 connection(s)
I20260929 21:00:25.853922 8992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
I20260929 21:00:25.854657 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 8992
I20260929 21:00:25.854832 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-2/wal/instance
I20260929 21:00:25.855386 388 external_mini_cluster.cc:1428] Setting key 38fc3500a4f2a58eddbe16c99b8c10e9
Sep 29 21:00:25 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:00:25.874662 8855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:25.857361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57145 (local address 127.0.97.62:39497)
0929 21:00:25.857550 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:25.857556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:25.859128 (+ 1572us) server_negotiation.cc:407] Connection header received
0929 21:00:25.861373 (+ 2245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:25.861377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:25.861437 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:25.861519 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:25.862868 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:25.863593 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:25.865091 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:25.865500 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:25.865590 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:25.868934 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:25.868938 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:25.868941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:25.868969 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:25.872169 (+ 3200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:25.872809 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:25.872813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:25.872815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:25.872978 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:25.873323 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:25.873326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:25.873328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:25.873570 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:25.873677 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:25.874277 (+ 600us) server_negotiation.cc:299] Negotiation successful
0929 21:00:25.874428 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 21:00:25.875568 9122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:25.858109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.3:57145)
0929 21:00:25.858964 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:25.859018 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:25.861135 (+ 2117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:25.861702 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:25.861709 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:25.862201 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:25.862731 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:25.862747 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:25.863713 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:25.863716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:25.864976 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:25.864988 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:25.865680 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:25.865685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:25.865863 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:25.866556 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:25.866578 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:25.868717 (+ 2139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:25.872283 (+ 3566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:25.872290 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:25.872304 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:25.872714 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:25.873071 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:25.873074 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:25.873076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:25.873235 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:25.873677 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:25.873682 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:25.873966 (+ 284us) client_negotiation.cc:769] Sending connection context
0929 21:00:25.874184 (+ 218us) client_negotiation.cc:240] Negotiation successful
0929 21:00:25.874536 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":333,"threads_started":1}
I20260929 21:00:25.876895 9120 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39497
I20260929 21:00:25.877198 9120 heartbeater.cc:464] Registering TS with master...
I20260929 21:00:25.877862 9120 heartbeater.cc:511] Master 127.0.97.62:39497 requested a full tablet report, sending...
I20260929 21:00:25.879212 8664 ts_manager.cc:194] Registered new tserver with Master: 83207e7121784978b030c97faeeb997f (127.0.97.3:42169)
I20260929 21:00:25.880059 8664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57145
I20260929 21:00:25.883320 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 21:00:25.889073 8853 heartbeater.cc:503] Master 127.0.97.62:39497 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260929 21:00:26.013151 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 8725
Sep 29 21:00:26 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 21:00:26.028175 9130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:26.014144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40832 (local address 127.0.97.1:43645)
0929 21:00:26.014749 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:26.014757 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:26.014785 (+ 28us) server_negotiation.cc:407] Connection header received
0929 21:00:26.014851 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:26.014858 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:26.015073 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:26.015209 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:26.016485 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:26.017553 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:26.018724 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:26.019273 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:26.019369 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:26.023154 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:26.023164 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:26.023178 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:26.023211 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:26.025991 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:26.026565 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:26.026571 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:26.026573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:26.026681 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:26.026974 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:26.026978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:26.026979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:26.027283 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:26.027499 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:26.027701 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 21:00:26.027897 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":302,"threads_started":1}
I20260929 21:00:26.029474 8848 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:00:26.111850 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 8725
I20260929 21:00:26.128543 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 8859
Sep 29 21:00:26 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 21:00:26.144315 9132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:26.129476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60004 (local address 127.0.97.2:44387)
0929 21:00:26.129954 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:26.129961 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:26.129984 (+ 23us) server_negotiation.cc:407] Connection header received
0929 21:00:26.130040 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:26.130046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:26.130251 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:26.130422 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:26.131607 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:26.132580 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:26.133872 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:26.134416 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:26.134550 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:26.138768 (+ 4218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:26.138778 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:26.138793 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:26.138825 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:26.141601 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:26.142255 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:26.142262 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:26.142264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:26.142379 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:26.142754 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:26.142762 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:26.142764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:26.143083 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:26.143392 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:26.143701 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 21:00:26.143963 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":234,"threads_started":1}
I20260929 21:00:26.145618 8982 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:00:26.226903 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 8859
I20260929 21:00:26.242715 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 8992
Sep 29 21:00:26 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 21:00:26.257891 9134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:26.243646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42278 (local address 127.0.97.3:42169)
0929 21:00:26.244200 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:26.244207 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:26.244234 (+ 27us) server_negotiation.cc:407] Connection header received
0929 21:00:26.244294 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:26.244301 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:26.244513 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:26.244643 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:26.245873 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:26.246866 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:26.248074 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:26.248606 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:26.248698 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:26.252982 (+ 4284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:26.252993 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:26.253008 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:26.253044 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:26.255597 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:26.256165 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:26.256172 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:26.256174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:26.256287 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:26.256584 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:26.256587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:26.256589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:26.256981 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:26.257237 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:26.257418 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 21:00:26.257606 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":273,"threads_started":1}
I20260929 21:00:26.259142 9115 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:00:26.341528 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 8992
I20260929 21:00:26.357033 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 8632
I20260929 21:00:26.358146 8694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:00:26.459470 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 8632
I20260929 21:00:26.480266 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39497
--webserver_interface=127.0.97.62
--webserver_port=43615
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39497
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:00:26.795281 9137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:26.795547 9137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:26.795652 9137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:26.802700 9137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:00:26.802803 9137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:26.802914 9137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:26.803014 9137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:00:26.803103 9137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:00:26.812630 9137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=43615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:00:26.813917 9137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:26.815186 9137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:26.826613 9143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:26.826625 9142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:26.826784 9145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:26.827241 9137 server_base.cc:1034] running on GCE node
I20260929 21:00:26.828009 9137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:26.829449 9137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:26.830668 9137 hybrid_clock.cc:648] HybridClock initialized: now 1790715626830622 us; error 37 us; skew 500 ppm
I20260929 21:00:26.831012 9137 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:00:26 dist-test-slave-1dms krb5kdc[8619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:00:26.833979 9137 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 21:00:26.835320 9137 webserver.cc:460] Webserver started at http://127.0.97.62:43615/ using document root <none> and password file <none>
I20260929 21:00:26.835978 9137 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:26.836135 9137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:26.841271 9137 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 21:00:26.843986 9152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:26.845368 9137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:00:26.845534 9137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "c75b9414215e432c84a48581b0f3bad3"
format_stamp: "Formatted at 2026-09-29 21:00:24 on dist-test-slave-1dms"
server_key: "32aa2ce6f64d759e28f0e9be3d5e0829"
server_key_iv: "eff210887a351b2bd8f9d0427f24a24f"
server_key_version: "encryptionkey@0"
I20260929 21:00:26.846455 9137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:26.861100 9137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:26.865031 9137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:00:26.865298 9137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:26.883337 9137 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39497
I20260929 21:00:26.883369 9203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39497 every 8 connection(s)
I20260929 21:00:26.884811 9137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:00:26.890942 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 9137
I20260929 21:00:26.891958 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:43645
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:39497
--webserver_port=42483
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 21:00:26.892076 9204 sys_catalog.cc:263] Verifying existing consensus state
I20260929 21:00:26.897199 9204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3: Bootstrap starting.
I20260929 21:00:26.916327 9204 log.cc:824] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3: Log is configured to *not* fsync() on all Append() calls
I20260929 21:00:26.924793 9204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 21:00:26.925352 9204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3: Bootstrap complete.
I20260929 21:00:26.934134 9204 raft_consensus.cc:348] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c75b9414215e432c84a48581b0f3bad3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39497 } }
I20260929 21:00:26.934949 9204 raft_consensus.cc:729] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c75b9414215e432c84a48581b0f3bad3, State: Initialized, Role: FOLLOWER
I20260929 21:00:26.935505 9204 consensus_queue.cc:260] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c75b9414215e432c84a48581b0f3bad3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39497 } }
I20260929 21:00:26.935740 9204 raft_consensus.cc:388] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:00:26.935863 9204 raft_consensus.cc:482] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:00:26.936040 9204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3 [term 1 FOLLOWER]: Advancing to term 2
I20260929 21:00:26.939785 9204 raft_consensus.cc:504] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c75b9414215e432c84a48581b0f3bad3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39497 } }
I20260929 21:00:26.940279 9204 leader_election.cc:302] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c75b9414215e432c84a48581b0f3bad3; no voters:
I20260929 21:00:26.940930 9204 leader_election.cc:288] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 21:00:26.941133 9208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 21:00:26.942131 9208 raft_consensus.cc:686] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3 [term 2 LEADER]: Becoming Leader. State: Replica: c75b9414215e432c84a48581b0f3bad3, State: Running, Role: LEADER
I20260929 21:00:26.942922 9204 sys_catalog.cc:564] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:00:26.942813 9208 consensus_queue.cc:237] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c75b9414215e432c84a48581b0f3bad3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39497 } }
I20260929 21:00:26.945243 9209 sys_catalog.cc:455] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c75b9414215e432c84a48581b0f3bad3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c75b9414215e432c84a48581b0f3bad3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39497 } } }
I20260929 21:00:26.945576 9209 sys_catalog.cc:458] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3 [sys.catalog]: This master's current role is: LEADER
I20260929 21:00:26.946424 9210 sys_catalog.cc:455] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c75b9414215e432c84a48581b0f3bad3. Latest consensus state: current_term: 2 leader_uuid: "c75b9414215e432c84a48581b0f3bad3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c75b9414215e432c84a48581b0f3bad3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39497 } } }
I20260929 21:00:26.946766 9210 sys_catalog.cc:458] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3 [sys.catalog]: This master's current role is: LEADER
I20260929 21:00:26.949244 9213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:00:26.955945 9213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:00:26.957306 9213 catalog_manager.cc:1259] Loaded cluster ID: 589fc8135e094ae389b68848ed86b316
I20260929 21:00:26.957454 9213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:00:26.961308 9213 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:00:26.962918 9213 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c75b9414215e432c84a48581b0f3bad3: Loaded TSK: 0
I20260929 21:00:26.964460 9213 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 21:00:27.226025 9206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:27.226279 9206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:27.226423 9206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:27.233711 9206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:27.233816 9206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:27.233980 9206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 21:00:27.243589 9206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:43645
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=42483
--tserver_master_addrs=127.0.97.62:39497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:00:27.245057 9206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:27.246325 9206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:27.261157 9230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:27.261186 9233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:27.261713 9231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:27.262542 9206 server_base.cc:1034] running on GCE node
I20260929 21:00:27.263211 9206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:27.264156 9206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:27.265383 9206 hybrid_clock.cc:648] HybridClock initialized: now 1790715627265318 us; error 59 us; skew 500 ppm
I20260929 21:00:27.265707 9206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:00:27 dist-test-slave-1dms krb5kdc[8619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715627, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:00:27.269057 9206 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260929 21:00:27.270722 9206 webserver.cc:460] Webserver started at http://127.0.97.1:42483/ using document root <none> and password file <none>
I20260929 21:00:27.271400 9206 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:27.271559 9206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:27.276970 9206 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:00:27.279678 9240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:27.281106 9206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:00:27.281258 9206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-0/wal
uuid: "24fa043915444d9199aaac5ddae48217"
format_stamp: "Formatted at 2026-09-29 21:00:24 on dist-test-slave-1dms"
server_key: "e2221f930d1eeec6473de3708a7e3dce"
server_key_iv: "b3c98884de0c8c583a742c32584f7960"
server_key_version: "encryptionkey@0"
I20260929 21:00:27.282160 9206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:27.302475 9206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:27.306564 9206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:00:27.306838 9206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:27.307773 9206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:00:27.309115 9206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:00:27.309201 9206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:27.309351 9206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:00:27.309453 9206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:27.341729 9206 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43645
I20260929 21:00:27.341820 9352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43645 every 8 connection(s)
I20260929 21:00:27.343246 9206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
I20260929 21:00:27.345089 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 9206
I20260929 21:00:27.345870 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:44387
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:39497
--webserver_port=43993
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 21:00:27 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:27 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 21:00:27.367398 9356 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:27.346500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45435 (local address 127.0.97.62:39497)
0929 21:00:27.353238 (+ 6738us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:27.353250 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:27.353290 (+ 40us) server_negotiation.cc:407] Connection header received
0929 21:00:27.353565 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:27.353577 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:27.353909 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:27.354263 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:27.355716 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:27.357793 (+ 2077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:27.359375 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:27.360502 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:27.360702 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:27.366515 (+ 5813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45435: BlockingRecv error: recv got EOF from 127.0.97.1:45435 (error 108)
Metrics: {"server-negotiator.queue_time_us":6534,"thread_start_us":6374,"threads_started":1}
I20260929 21:00:27.367563 9355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:27.346512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.1:45435)
0929 21:00:27.347312 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:27.347357 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:27.349106 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:27.354423 (+ 5317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:27.354435 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:27.355011 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:27.355526 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:27.355541 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:27.357965 (+ 2424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:27.357970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:27.359224 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:27.359237 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:27.360730 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:27.360735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:27.360992 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:27.361710 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:27.361737 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:27.366222 (+ 4485us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":347,"threads_started":1}
W20260929 21:00:27.368464 9353 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39497 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:39497: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 29 21:00:27 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:27 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:27.378196 9355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:27.369110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.1:41085)
0929 21:00:27.369293 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:27.369314 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:27.369429 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:27.369812 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:27.369820 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:27.370301 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:27.370512 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:27.370527 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:27.371592 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:27.371599 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:27.373296 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:27.373310 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:27.373958 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:27.373963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:27.374135 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:27.374700 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:27.374722 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:27.377921 (+ 3199us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 21:00:27.378422 9356 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:27.369221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41085 (local address 127.0.97.62:39497)
0929 21:00:27.369355 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:27.369359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:27.369374 (+ 15us) server_negotiation.cc:407] Connection header received
0929 21:00:27.369581 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:27.369585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:27.369646 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:27.369748 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:27.370713 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:27.371441 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:27.373435 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:27.373825 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:27.373919 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:27.378101 (+ 4182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:41085: BlockingRecv error: recv got EOF from 127.0.97.1:41085 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 29 21:00:27 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:27 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:27.387266 9355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:27.378795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.1:51695)
0929 21:00:27.378961 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:27.378979 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:27.379067 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:27.379425 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:27.379430 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:27.379856 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:27.380054 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:27.380064 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:27.381036 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:27.381041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:27.382528 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:27.382542 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:27.383183 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:27.383189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:27.383356 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:27.383908 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:27.383931 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:27.386969 (+ 3038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 21:00:27.387394 9356 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:27.379037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51695 (local address 127.0.97.62:39497)
0929 21:00:27.379185 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:27.379189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:27.379203 (+ 14us) server_negotiation.cc:407] Connection header received
0929 21:00:27.379239 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:27.379243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:27.379304 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:27.379379 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:27.380182 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:27.380902 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:27.382664 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:27.383051 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:27.383145 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:27.387144 (+ 3999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:51695: BlockingRecv error: recv got EOF from 127.0.97.1:51695 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 21:00:27.387576 9353 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 21:00:27.675570 9357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:27.675885 9357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:27.676048 9357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:27.683080 9357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:27.683223 9357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:27.683404 9357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 21:00:27.692873 9357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:44387
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=43993
--tserver_master_addrs=127.0.97.62:39497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:00:27.694175 9357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:27.695446 9357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:27.708161 9363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:27.708123 9365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:27.708913 9357 server_base.cc:1034] running on GCE node
W20260929 21:00:27.709000 9362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:27.709606 9357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:27.710515 9357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:27.711796 9357 hybrid_clock.cc:648] HybridClock initialized: now 1790715627711746 us; error 53 us; skew 500 ppm
I20260929 21:00:27.712138 9357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:00:27 dist-test-slave-1dms krb5kdc[8619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715627, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:00:27.715162 9357 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260929 21:00:27.716424 9357 webserver.cc:460] Webserver started at http://127.0.97.2:43993/ using document root <none> and password file <none>
I20260929 21:00:27.717104 9357 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:27.717209 9357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:27.722024 9357 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 21:00:27.724417 9372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:27.725694 9357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:00:27.725834 9357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-1/wal
uuid: "67c1353efcd249daa78705582ab51a13"
format_stamp: "Formatted at 2026-09-29 21:00:25 on dist-test-slave-1dms"
server_key: "3ada611c76ab6535f5dd62511b7ffb4b"
server_key_iv: "0e24adc29d9172fc0507dbf4eb8f0095"
server_key_version: "encryptionkey@0"
I20260929 21:00:27.726670 9357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:27.740434 9357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:27.743911 9357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:00:27.744115 9357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:27.744966 9357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:00:27.746125 9357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:00:27.746205 9357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:27.746276 9357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:00:27.746330 9357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:27.777768 9357 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44387
I20260929 21:00:27.777814 9484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44387 every 8 connection(s)
I20260929 21:00:27.779168 9357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
I20260929 21:00:27.781126 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 9357
I20260929 21:00:27.781787 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:42169
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:39497
--webserver_port=35789
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 21:00:27 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:27 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 21:00:27.795002 9356 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:27.781650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60971 (local address 127.0.97.62:39497)
0929 21:00:27.781849 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:27.781855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:27.783106 (+ 1251us) server_negotiation.cc:407] Connection header received
0929 21:00:27.784947 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:27.784955 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:27.785063 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:27.785177 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:27.786665 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:27.787700 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:27.789606 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:27.790026 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:27.790160 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:27.794745 (+ 4585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:60971: BlockingRecv error: recv got EOF from 127.0.97.2:60971 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 21:00:27.795665 9487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:27.782164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.2:60971)
0929 21:00:27.782934 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:27.782978 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:27.784659 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:27.785367 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:27.785374 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:27.785923 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:27.786502 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:27.786520 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:27.787855 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:27.787862 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:27.789469 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:27.789480 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:27.790261 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:27.790268 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:27.790524 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:27.791190 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:27.791211 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:27.794585 (+ 3374us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":325,"threads_started":1}
W20260929 21:00:27.796303 9485 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39497 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:39497: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 29 21:00:27 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:27 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:27.805015 9487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:27.796934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.2:49637)
0929 21:00:27.797080 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:27.797099 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:27.797191 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:27.797541 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:27.797544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:27.797806 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:27.797933 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:27.797939 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:27.799417 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:27.799421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:27.800263 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:27.800272 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:27.801340 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:27.801344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:27.801456 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:27.802075 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:27.802092 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:27.804728 (+ 2636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260929 21:00:27.805366 9356 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:27.797063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49637 (local address 127.0.97.62:39497)
0929 21:00:27.797239 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:27.797247 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:27.797268 (+ 21us) server_negotiation.cc:407] Connection header received
0929 21:00:27.797322 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:27.797330 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:27.797412 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:27.797523 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:27.798065 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:27.799271 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:27.800410 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:27.801203 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:27.801350 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:27.804941 (+ 3591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:49637: BlockingRecv error: recv got EOF from 127.0.97.2:49637 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 29 21:00:27 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:27 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:27.813691 9487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:27.805618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.2:58047)
0929 21:00:27.805869 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:27.805886 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:27.805970 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:27.806277 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:27.806281 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:27.806619 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:27.806794 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:27.806804 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:27.807696 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:27.807701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:27.809176 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:27.809192 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:27.809830 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:27.809836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:27.810006 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:27.810528 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:27.810549 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:27.813413 (+ 2864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152}
W20260929 21:00:27.813796 9356 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:27.805864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58047 (local address 127.0.97.62:39497)
0929 21:00:27.806017 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:27.806021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:27.806034 (+ 13us) server_negotiation.cc:407] Connection header received
0929 21:00:27.806103 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:27.806106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:27.806158 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:27.806231 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:27.806913 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:27.807572 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:27.809305 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:27.809687 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:27.809774 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:27.813593 (+ 3819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58047: BlockingRecv error: recv got EOF from 127.0.97.2:58047 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 21:00:27.814035 9485 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 21:00:28.101313 9488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:28.101562 9488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:28.101655 9488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:28.108735 9488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:28.108872 9488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:28.109007 9488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 21:00:28.118219 9488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:42169
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=35789
--tserver_master_addrs=127.0.97.62:39497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:00:28.119486 9488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:28.120766 9488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:28.133463 9496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:28.133514 9494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:28.133558 9493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:28.134390 9488 server_base.cc:1034] running on GCE node
I20260929 21:00:28.134958 9488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:28.135840 9488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:28.137058 9488 hybrid_clock.cc:648] HybridClock initialized: now 1790715628137014 us; error 53 us; skew 500 ppm
I20260929 21:00:28.137352 9488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:00:28 dist-test-slave-1dms krb5kdc[8619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715628, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:00:28.140427 9488 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260929 21:00:28.141788 9488 webserver.cc:460] Webserver started at http://127.0.97.3:35789/ using document root <none> and password file <none>
I20260929 21:00:28.142453 9488 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:28.142604 9488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:28.147653 9488 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 21:00:28.150421 9503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:28.151777 9488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 21:00:28.151921 9488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-2/wal
uuid: "83207e7121784978b030c97faeeb997f"
format_stamp: "Formatted at 2026-09-29 21:00:25 on dist-test-slave-1dms"
server_key: "12d61f2a8ed88fa4f7943ce3b1a63ac3"
server_key_iv: "64e369750287b3b7ddc5e5248f861f0e"
server_key_version: "encryptionkey@0"
I20260929 21:00:28.152778 9488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:28.170897 9488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:28.174929 9488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:00:28.175148 9488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:28.176033 9488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:00:28.177312 9488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:00:28.177395 9488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:28.177481 9488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:00:28.177537 9488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:28.209965 9488 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42169
I20260929 21:00:28.210058 9615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42169 every 8 connection(s)
I20260929 21:00:28.211396 9488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
I20260929 21:00:28.212080 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 9488
Sep 29 21:00:28 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:28 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:28.228307 9356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:28.213113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38994 (local address 127.0.97.62:39497)
0929 21:00:28.213313 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:28.213319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:28.213827 (+ 508us) server_negotiation.cc:407] Connection header received
0929 21:00:28.213977 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:28.213984 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:28.214068 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:28.214165 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:28.215912 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:28.216594 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:28.218414 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:28.218777 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:28.218880 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:28.221682 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:28.221689 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:28.221702 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:28.221734 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:28.224934 (+ 3200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:28.225581 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:28.225588 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:28.225594 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:28.225839 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:28.226227 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:28.226232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:28.226235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:28.226820 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:28.227059 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:28.227550 (+ 491us) server_negotiation.cc:299] Negotiation successful
0929 21:00:28.227881 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 21:00:28.229342 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:28.214210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47823 (local address 127.0.97.62:39497)
0929 21:00:28.214631 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:28.214636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:28.215844 (+ 1208us) server_negotiation.cc:407] Connection header received
0929 21:00:28.218104 (+ 2260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:28.218108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:28.218162 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:28.218236 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:28.220082 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:28.220786 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:28.222758 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:28.223494 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:28.223632 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:28.229083 (+ 5451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:47823: BlockingRecv error: recv got EOF from 127.0.97.3:47823 (error 108)
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":240,"threads_started":1}
I20260929 21:00:28.229877 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:28.214846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.3:47823)
0929 21:00:28.215659 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:28.215720 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:28.217813 (+ 2093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:28.218511 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:28.218518 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:28.219137 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:28.219891 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:28.219928 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:28.220983 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:28.220989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:28.222625 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:28.222636 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:28.223764 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:28.223773 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:28.224064 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:28.224635 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:28.224659 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:28.228763 (+ 4104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":302,"threads_started":1}
W20260929 21:00:28.230600 9616 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39497 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:39497: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 29 21:00:28 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:28 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:28.238508 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:28.231411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.3:34793)
0929 21:00:28.231949 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:28.231965 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:28.232034 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:28.232382 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:28.232385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:28.232644 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:28.232762 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:28.232768 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:28.233741 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:28.233744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:28.234587 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:28.234596 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:28.235172 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:28.235176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:28.235286 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:28.235732 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:28.235747 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:28.238222 (+ 2475us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420}
W20260929 21:00:28.238610 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:28.231458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34793 (local address 127.0.97.62:39497)
0929 21:00:28.232173 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:28.232177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:28.232188 (+ 11us) server_negotiation.cc:407] Connection header received
0929 21:00:28.232220 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:28.232223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:28.232267 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:28.232333 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:28.232928 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:28.233628 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:28.234696 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:28.235064 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:28.235148 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:28.238396 (+ 3248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:34793: BlockingRecv error: recv got EOF from 127.0.97.3:34793 (error 108)
Metrics: {"server-negotiator.queue_time_us":639}
Sep 29 21:00:28 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:28 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:28.246623 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:28.239164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.3:42243)
0929 21:00:28.239304 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:28.239348 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:28.239438 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:28.239702 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:28.239706 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:28.239988 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:28.240144 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:28.240155 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:28.241141 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:28.241147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:28.242024 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:28.242034 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:28.242801 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:28.242807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:28.243029 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:28.243639 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:28.243666 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:28.246434 (+ 2768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 21:00:28.246899 9616 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 21:00:28.247084 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:28.239226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42243 (local address 127.0.97.62:39497)
0929 21:00:28.239454 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:28.239459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:28.239478 (+ 19us) server_negotiation.cc:407] Connection header received
0929 21:00:28.239526 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:28.239530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:28.239586 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:28.239681 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:28.240288 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:28.241014 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:28.242171 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:28.242658 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:28.242771 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:28.246573 (+ 3802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:42243: BlockingRecv error: recv got EOF from 127.0.97.3:42243 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
Sep 29 21:00:28 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:28 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:28.396601 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:28.388143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.1:46019)
0929 21:00:28.388638 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:28.388654 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:28.388766 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:28.389104 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:28.389107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:28.389542 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:28.389736 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:28.389745 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:28.390735 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:28.390738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:28.391680 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:28.391691 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:28.392460 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:28.392464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:28.392580 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:28.393158 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:28.393175 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:28.396382 (+ 3207us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":280,"threads_started":1}
W20260929 21:00:28.396914 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:28.388186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46019 (local address 127.0.97.62:39497)
0929 21:00:28.388422 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:28.388432 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:28.388738 (+ 306us) server_negotiation.cc:407] Connection header received
0929 21:00:28.388902 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:28.388906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:28.388980 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:28.389066 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:28.389870 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:28.390619 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:28.391823 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:28.392297 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:28.392400 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:28.396588 (+ 4188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:46019: BlockingRecv error: recv got EOF from 127.0.97.1:46019 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 29 21:00:28 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:28 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:28.822705 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:28.814680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.2:44017)
0929 21:00:28.815143 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:28.815159 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:28.815264 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:28.815529 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:28.815532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:28.815961 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:28.816151 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:28.816158 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:28.817085 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:28.817089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:28.818029 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:28.818039 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:28.818722 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:28.818726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:28.818876 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:28.819609 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:28.819626 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:28.822497 (+ 2871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":263,"threads_started":1}
W20260929 21:00:28.822911 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:28.814736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44017 (local address 127.0.97.62:39497)
0929 21:00:28.814945 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:28.814954 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:28.815237 (+ 283us) server_negotiation.cc:407] Connection header received
0929 21:00:28.815352 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:28.815355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:28.815411 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:28.815487 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:28.816263 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:28.816976 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:28.818142 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:28.818537 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:28.818631 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:28.822667 (+ 4036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:44017: BlockingRecv error: recv got EOF from 127.0.97.2:44017 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 21:00:29 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:29 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:29.264222 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:29.247406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.3:35197)
0929 21:00:29.247906 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:29.247927 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:29.248036 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:29.248303 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:29.248306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:29.248737 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:29.248991 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:29.249001 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:29.250032 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:29.250035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:29.251064 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:29.251076 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:29.251835 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:29.251840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:29.251962 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:29.261159 (+ 9197us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:29.261178 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:29.263943 (+ 2765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":272,"threads_started":1}
W20260929 21:00:29.264343 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:29.247492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35197 (local address 127.0.97.62:39497)
0929 21:00:29.247647 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:29.247651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:29.247998 (+ 347us) server_negotiation.cc:407] Connection header received
0929 21:00:29.248130 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:29.248133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:29.248189 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:29.248259 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:29.249132 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:29.249921 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:29.251181 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:29.251684 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:29.251776 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:29.264132 (+ 12356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:35197: BlockingRecv error: recv got EOF from 127.0.97.3:35197 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 21:00:29 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:29 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:29.404904 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:29.397391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.1:42963)
0929 21:00:29.397918 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:29.397934 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:29.398048 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:29.398360 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:29.398363 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:29.398736 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:29.398872 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:29.398880 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:29.399795 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:29.399798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:29.400689 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:29.400701 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:29.401341 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:29.401345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:29.401456 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:29.402005 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:29.402022 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:29.404611 (+ 2589us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":295,"threads_started":1}
W20260929 21:00:29.405131 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:29.397466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42963 (local address 127.0.97.62:39497)
0929 21:00:29.397624 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:29.397628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:29.398007 (+ 379us) server_negotiation.cc:407] Connection header received
0929 21:00:29.398155 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:29.398159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:29.398227 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:29.398304 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:29.399000 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:29.399682 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:29.400807 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:29.401215 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:29.401301 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:29.404776 (+ 3475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:42963: BlockingRecv error: recv got EOF from 127.0.97.1:42963 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 21:00:29 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:29 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:29.832096 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:29.823539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.2:60729)
0929 21:00:29.824040 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:29.824054 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:29.824208 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:29.824531 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:29.824536 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:29.825023 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:29.825165 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:29.825173 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:29.826157 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:29.826162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:29.827224 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:29.827235 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:29.827932 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:29.827938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:29.828102 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:29.828654 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:29.828672 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:29.831870 (+ 3198us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":276,"threads_started":1}
W20260929 21:00:29.832243 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:29.823631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60729 (local address 127.0.97.62:39497)
0929 21:00:29.823872 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:29.823881 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:29.824141 (+ 260us) server_negotiation.cc:407] Connection header received
0929 21:00:29.824326 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:29.824331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:29.824396 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:29.824508 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:29.825304 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:29.826021 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:29.827369 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:29.827779 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:29.827907 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:29.832053 (+ 4146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:60729: BlockingRecv error: recv got EOF from 127.0.97.2:60729 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 29 21:00:30 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:30 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:30.273409 9626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:30.265273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.3:45871)
0929 21:00:30.265798 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:30.265821 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:30.265940 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:30.266270 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:30.266276 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:30.266789 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:30.266967 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:30.266976 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:30.268011 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:30.268017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:30.268995 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:30.269006 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:30.269791 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:30.269797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:30.269965 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:30.270475 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:30.270490 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:30.273074 (+ 2584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":292,"threads_started":1}
W20260929 21:00:30.273515 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:30.265420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45871 (local address 127.0.97.62:39497)
0929 21:00:30.265573 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:30.265577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:30.265889 (+ 312us) server_negotiation.cc:407] Connection header received
0929 21:00:30.266049 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:30.266052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:30.266110 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:30.266224 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:30.267103 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:30.267867 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:30.269127 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:30.269618 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:30.269782 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:30.273260 (+ 3478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:45871: BlockingRecv error: recv got EOF from 127.0.97.3:45871 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 21:00:30 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:30 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:30.413980 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:30.405712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.1:39949)
0929 21:00:30.406236 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:30.406251 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:30.406412 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:30.406733 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:30.406737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:30.407100 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:30.407322 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:30.407344 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:30.408790 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:30.408797 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:30.409783 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:30.409794 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:30.410400 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:30.410403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:30.410519 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:30.411077 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:30.411094 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:30.413741 (+ 2647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":281,"threads_started":1}
W20260929 21:00:30.414147 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:30.405891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39949 (local address 127.0.97.62:39497)
0929 21:00:30.406118 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:30.406124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:30.406328 (+ 204us) server_negotiation.cc:407] Connection header received
0929 21:00:30.406545 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:30.406549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:30.406606 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:30.406703 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:30.407506 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:30.408634 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:30.409914 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:30.410278 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:30.410373 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:30.413943 (+ 3570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:39949: BlockingRecv error: recv got EOF from 127.0.97.1:39949 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 29 21:00:30 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:30 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:30.841038 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:30.833065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.2:51169)
0929 21:00:30.833600 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:30.833617 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:30.833731 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:30.834010 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:30.834014 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:30.834327 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:30.834481 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:30.834488 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:30.835433 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:30.835436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:30.836306 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:30.836316 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:30.837261 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:30.837265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:30.837392 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:30.837977 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:30.837994 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:30.840712 (+ 2718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":314,"threads_started":1}
W20260929 21:00:30.841073 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:30.833112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51169 (local address 127.0.97.62:39497)
0929 21:00:30.833278 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:30.833283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:30.833703 (+ 420us) server_negotiation.cc:407] Connection header received
0929 21:00:30.833826 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:30.833829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:30.833889 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:30.833972 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:30.834594 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:30.835305 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:30.836443 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:30.837137 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:30.837243 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:30.840895 (+ 3652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51169: BlockingRecv error: recv got EOF from 127.0.97.2:51169 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 21:00:31 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:31 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:31.282091 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:31.274409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.3:35369)
0929 21:00:31.274923 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:31.274954 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:31.275066 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:31.275338 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:31.275341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:31.275685 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:31.275841 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:31.275849 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:31.276795 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:31.276798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:31.277699 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:31.277710 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:31.278344 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:31.278348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:31.278477 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:31.279142 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:31.279158 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:31.281846 (+ 2688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":283,"threads_started":1}
W20260929 21:00:31.282220 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:31.274449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35369 (local address 127.0.97.62:39497)
0929 21:00:31.274633 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:31.274637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:31.275026 (+ 389us) server_negotiation.cc:407] Connection header received
0929 21:00:31.275156 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:31.275159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:31.275217 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:31.275295 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:31.275963 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:31.276684 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:31.277815 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:31.278210 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:31.278308 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:31.282030 (+ 3722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:35369: BlockingRecv error: recv got EOF from 127.0.97.3:35369 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 29 21:00:31 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:31 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:31.423198 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:31.414895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.1:33885)
0929 21:00:31.415362 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:31.415377 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:31.415554 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:31.415878 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:31.415882 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:31.416258 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:31.416415 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:31.416423 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:31.417513 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:31.417521 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:31.418711 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:31.418724 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:31.419454 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:31.419460 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:31.419643 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:31.420277 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:31.420294 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:31.422974 (+ 2680us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":258,"threads_started":1}
W20260929 21:00:31.423429 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:31.414964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33885 (local address 127.0.97.62:39497)
0929 21:00:31.415214 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:31.415220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:31.415477 (+ 257us) server_negotiation.cc:407] Connection header received
0929 21:00:31.415678 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:31.415683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:31.415753 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:31.415853 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:31.416558 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:31.417387 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:31.418865 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:31.419302 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:31.419444 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:31.423188 (+ 3744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:33885: BlockingRecv error: recv got EOF from 127.0.97.1:33885 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Sep 29 21:00:31 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:31 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:31.850106 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:31.841972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.2:41627)
0929 21:00:31.842426 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:31.842442 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:31.842590 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:31.842921 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:31.842925 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:31.843295 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:31.843487 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:31.843497 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:31.844516 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:31.844523 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:31.845483 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:31.845494 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:31.846227 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:31.846233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:31.846416 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:31.846970 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:31.846988 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:31.849774 (+ 2786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":235,"threads_started":1}
W20260929 21:00:31.850207 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:31.841993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41627 (local address 127.0.97.62:39497)
0929 21:00:31.842224 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:31.842233 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:31.842519 (+ 286us) server_negotiation.cc:407] Connection header received
0929 21:00:31.842713 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:31.842718 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:31.842782 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:31.842897 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:31.843622 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:31.844391 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:31.845620 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:31.846077 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:31.846203 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:31.849960 (+ 3757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:41627: BlockingRecv error: recv got EOF from 127.0.97.2:41627 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 21:00:32 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:32 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:32.291198 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:32.283037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.3:58433)
0929 21:00:32.283609 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:32.283629 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:32.283742 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:32.284031 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:32.284035 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:32.284400 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:32.284541 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:32.284549 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:32.285528 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:32.285531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:32.286572 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:32.286583 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:32.287213 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:32.287217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:32.287345 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:32.287949 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:32.287966 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:32.290969 (+ 3003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":307,"threads_started":1}
W20260929 21:00:32.291363 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:32.283116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58433 (local address 127.0.97.62:39497)
0929 21:00:32.283288 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:32.283293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:32.283704 (+ 411us) server_negotiation.cc:407] Connection header received
0929 21:00:32.283849 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:32.283853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:32.283912 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:32.283991 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:32.284657 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:32.285419 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:32.286701 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:32.287085 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:32.287176 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:32.291128 (+ 3952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58433: BlockingRecv error: recv got EOF from 127.0.97.3:58433 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 21:00:32 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:32 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:32.432705 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:32.424230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.1:33053)
0929 21:00:32.424740 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:32.424756 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:32.424937 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:32.425264 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:32.425269 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:32.425705 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:32.425886 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:32.425895 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:32.426897 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:32.426905 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:32.427800 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:32.427811 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:32.428484 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:32.428489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:32.428685 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:32.429421 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:32.429438 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:32.432361 (+ 2923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":257,"threads_started":1}
W20260929 21:00:32.432767 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:32.424305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33053 (local address 127.0.97.62:39497)
0929 21:00:32.424553 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:32.424559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:32.424879 (+ 320us) server_negotiation.cc:407] Connection header received
0929 21:00:32.425060 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:32.425067 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:32.425132 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:32.425245 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:32.426026 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:32.426755 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:32.427945 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:32.428343 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:32.428474 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:32.432564 (+ 4090us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:33053: BlockingRecv error: recv got EOF from 127.0.97.1:33053 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 21:00:32 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:32 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:32.859591 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:32.851152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.2:35787)
0929 21:00:32.851667 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:32.851683 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:32.851830 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:32.852171 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:32.852176 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:32.852594 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:32.852744 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:32.852752 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:32.853802 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:32.853809 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:32.854710 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:32.854721 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:32.855434 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:32.855440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:32.855609 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:32.856246 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:32.856264 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:32.859349 (+ 3085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":278,"threads_started":1}
W20260929 21:00:32.859719 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:32.851337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35787 (local address 127.0.97.62:39497)
0929 21:00:32.851536 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:32.851543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:32.851764 (+ 221us) server_negotiation.cc:407] Connection header received
0929 21:00:32.851963 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:32.851968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:32.852034 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:32.852144 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:32.852922 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:32.853672 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:32.854853 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:32.855261 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:32.855410 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:32.859520 (+ 4110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35787: BlockingRecv error: recv got EOF from 127.0.97.2:35787 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 29 21:00:33 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:33 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:33.300508 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:33.292264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.3:33989)
0929 21:00:33.292762 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:33.292782 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:33.292972 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:33.293299 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:33.293304 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:33.293712 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:33.293873 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:33.293882 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:33.294888 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:33.294896 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:33.295996 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:33.296007 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:33.296940 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:33.296946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:33.297073 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:33.297622 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:33.297638 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:33.300277 (+ 2639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":246,"threads_started":1}
W20260929 21:00:33.300702 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:33.292331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33989 (local address 127.0.97.62:39497)
0929 21:00:33.292550 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:33.292559 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:33.292887 (+ 328us) server_negotiation.cc:407] Connection header received
0929 21:00:33.293098 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:33.293109 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:33.293168 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:33.293271 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:33.294014 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:33.294746 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:33.296146 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:33.296769 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:33.296889 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:33.300450 (+ 3561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:33989: BlockingRecv error: recv got EOF from 127.0.97.3:33989 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 29 21:00:33 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:33 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:33.442034 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:33.433856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.1:40817)
0929 21:00:33.434378 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:33.434394 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:33.434504 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:33.434780 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:33.434783 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:33.435144 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:33.435406 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:33.435419 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:33.436590 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:33.436596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:33.437760 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:33.437773 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:33.438489 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:33.438493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:33.438617 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:33.439142 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:33.439159 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:33.441809 (+ 2650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":297,"threads_started":1}
W20260929 21:00:33.442231 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:33.433889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40817 (local address 127.0.97.62:39497)
0929 21:00:33.434107 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:33.434111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:33.434462 (+ 351us) server_negotiation.cc:407] Connection header received
0929 21:00:33.434595 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:33.434598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:33.434657 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:33.434740 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:33.435567 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:33.436392 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:33.437886 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:33.438359 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:33.438457 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:33.441982 (+ 3525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:40817: BlockingRecv error: recv got EOF from 127.0.97.1:40817 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 29 21:00:33 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:33 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:33.868625 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:33.860657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.2:34401)
0929 21:00:33.861231 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:33.861251 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:33.861394 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:33.861711 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:33.861715 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:33.862112 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:33.862256 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:33.862264 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:33.863234 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:33.863241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:33.864138 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:33.864149 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:33.864889 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:33.864896 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:33.865071 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:33.865720 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:33.865738 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:33.868404 (+ 2666us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":309,"threads_started":1}
W20260929 21:00:33.868763 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:33.860722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34401 (local address 127.0.97.62:39497)
0929 21:00:33.860953 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:33.860960 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:33.861316 (+ 356us) server_negotiation.cc:407] Connection header received
0929 21:00:33.861502 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:33.861508 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:33.861580 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:33.861680 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:33.862396 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:33.863100 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:33.864276 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:33.864710 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:33.864872 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:33.868575 (+ 3703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:34401: BlockingRecv error: recv got EOF from 127.0.97.2:34401 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 21:00:34 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:34 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:34.316042 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:34.301366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.3:58483)
0929 21:00:34.301904 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:34.301927 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:34.302053 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:34.302344 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:34.302348 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:34.302772 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:34.302926 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:34.302934 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:34.303946 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:34.303950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:34.305046 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:34.305059 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:34.305741 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:34.305746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:34.305872 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:34.312990 (+ 7118us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:34.313010 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:34.315724 (+ 2714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":322,"threads_started":1}
W20260929 21:00:34.316146 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:34.301550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58483 (local address 127.0.97.62:39497)
0929 21:00:34.301773 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:34.301779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:34.302008 (+ 229us) server_negotiation.cc:407] Connection header received
0929 21:00:34.302151 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:34.302154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:34.302215 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:34.302291 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:34.303046 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:34.303805 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:34.305173 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:34.305553 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:34.305640 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:34.315915 (+ 10275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58483: BlockingRecv error: recv got EOF from 127.0.97.3:58483 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 29 21:00:34 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:34 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:34.450855 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:34.442968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.1:59237)
0929 21:00:34.443533 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:34.443550 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:34.443663 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:34.443999 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:34.444003 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:34.444360 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:34.444573 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:34.444582 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:34.445539 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:34.445543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:34.446452 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:34.446463 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:34.447099 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:34.447103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:34.447210 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:34.447745 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:34.447762 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:34.450505 (+ 2743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":244,"threads_started":1}
W20260929 21:00:34.450933 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:34.443001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59237 (local address 127.0.97.62:39497)
0929 21:00:34.443234 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:34.443243 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:34.443626 (+ 383us) server_negotiation.cc:407] Connection header received
0929 21:00:34.443755 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:34.443759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:34.443818 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:34.443977 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:34.444687 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:34.445419 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:34.446565 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:34.446977 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:34.447072 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:34.450697 (+ 3625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:59237: BlockingRecv error: recv got EOF from 127.0.97.1:59237 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 21:00:34 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:34 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:34.877305 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:34.869638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.2:50447)
0929 21:00:34.870237 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:34.870254 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:34.870376 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:34.870650 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:34.870652 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:34.871011 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:34.871173 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:34.871181 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:34.872107 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:34.872110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:34.873085 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:34.873096 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:34.873750 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:34.873754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:34.873855 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:34.874372 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:34.874388 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:34.877068 (+ 2680us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":252,"threads_started":1}
W20260929 21:00:34.877524 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:34.869685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50447 (local address 127.0.97.62:39497)
0929 21:00:34.869893 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:34.869901 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:34.870371 (+ 470us) server_negotiation.cc:407] Connection header received
0929 21:00:34.870468 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:34.870472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:34.870531 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:34.870610 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:34.871290 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:34.871992 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:34.873204 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:34.873628 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:34.873718 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:34.877266 (+ 3548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:50447: BlockingRecv error: recv got EOF from 127.0.97.2:50447 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 29 21:00:35 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:35 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:35.325034 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:35.316984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.3:49407)
0929 21:00:35.317529 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:35.317549 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:35.317659 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:35.317935 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:35.317938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:35.318345 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:35.318513 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:35.318520 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:35.319473 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:35.319476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:35.320364 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:35.320374 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:35.321006 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:35.321010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:35.321110 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:35.321633 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:35.321650 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:35.324717 (+ 3067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":219,"threads_started":1}
W20260929 21:00:35.325133 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:35.317018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49407 (local address 127.0.97.62:39497)
0929 21:00:35.317225 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:35.317235 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:35.317624 (+ 389us) server_negotiation.cc:407] Connection header received
0929 21:00:35.317751 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:35.317754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:35.317814 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:35.317895 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:35.318630 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:35.319358 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:35.320480 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:35.320885 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:35.320981 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:35.324939 (+ 3958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:49407: BlockingRecv error: recv got EOF from 127.0.97.3:49407 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 21:00:35 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:35 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:35.460222 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:35.451730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.1:44117)
0929 21:00:35.452174 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:35.452190 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:35.452303 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:35.452611 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:35.452618 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:35.453211 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:35.453417 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:35.453432 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:35.454637 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:35.454640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:35.455528 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:35.455539 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:35.456243 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:35.456250 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:35.456388 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:35.457048 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:35.457065 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:35.459877 (+ 2812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":259,"threads_started":1}
W20260929 21:00:35.460316 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:35.451945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44117 (local address 127.0.97.62:39497)
0929 21:00:35.452171 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:35.452183 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:35.452269 (+ 86us) server_negotiation.cc:407] Connection header received
0929 21:00:35.452401 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:35.452404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:35.452469 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:35.452549 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:35.453561 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:35.454526 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:35.455671 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:35.456073 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:35.456164 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:35.460058 (+ 3894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:44117: BlockingRecv error: recv got EOF from 127.0.97.1:44117 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 21:00:35 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:35 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:35.887495 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:35.878248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.2:37835)
0929 21:00:35.878748 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:35.878764 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:35.878938 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:35.879295 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:35.879300 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:35.879746 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:35.879894 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:35.879903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:35.881513 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:35.881525 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:35.882525 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:35.882539 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:35.883257 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:35.883261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:35.883377 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:35.884025 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:35.884043 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:35.887190 (+ 3147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":289,"threads_started":1}
W20260929 21:00:35.887630 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:35.878338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37835 (local address 127.0.97.62:39497)
0929 21:00:35.878591 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:35.878599 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:35.878887 (+ 288us) server_negotiation.cc:407] Connection header received
0929 21:00:35.879072 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:35.879078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:35.879167 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:35.879269 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:35.880056 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:35.881367 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:35.882675 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:35.883105 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:35.883227 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:35.887397 (+ 4170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37835: BlockingRecv error: recv got EOF from 127.0.97.2:37835 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 29 21:00:36 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:36 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:36.334272 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:36.325971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.3:47185)
0929 21:00:36.326438 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:36.326459 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:36.326612 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:36.326952 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:36.326956 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:36.327358 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:36.327511 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:36.327520 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:36.328516 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:36.328522 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:36.329496 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:36.329508 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:36.330305 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:36.330313 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:36.330482 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:36.331128 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:36.331198 (+ 70us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:36.333949 (+ 2751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":248,"threads_started":1}
W20260929 21:00:36.334390 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:36.326033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47185 (local address 127.0.97.62:39497)
0929 21:00:36.326258 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:36.326266 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:36.326542 (+ 276us) server_negotiation.cc:407] Connection header received
0929 21:00:36.326740 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:36.326745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:36.326819 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:36.326929 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:36.327650 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:36.328377 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:36.329636 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:36.330120 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:36.330266 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:36.334158 (+ 3892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:47185: BlockingRecv error: recv got EOF from 127.0.97.3:47185 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 29 21:00:36 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:36 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:36.469415 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:36.461322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.1:48181)
0929 21:00:36.461867 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:36.461889 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:36.462035 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:36.462416 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:36.462421 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:36.462790 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:36.463002 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:36.463014 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:36.464118 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:36.464122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:36.465137 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:36.465148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:36.465753 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:36.465757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:36.465884 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:36.466434 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:36.466451 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:36.469164 (+ 2713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":260,"threads_started":1}
W20260929 21:00:36.469717 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:36.461392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48181 (local address 127.0.97.62:39497)
0929 21:00:36.461626 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:36.461634 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:36.461986 (+ 352us) server_negotiation.cc:407] Connection header received
0929 21:00:36.462175 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:36.462181 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:36.462269 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:36.462350 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:36.463164 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:36.463997 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:36.465251 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:36.465629 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:36.465732 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:36.469346 (+ 3614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48181: BlockingRecv error: recv got EOF from 127.0.97.1:48181 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 21:00:36 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:36 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:36.896523 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:36.888564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.2:34697)
0929 21:00:36.889110 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:36.889131 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:36.889260 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:36.889557 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:36.889562 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:36.890052 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:36.890267 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:36.890278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:36.891420 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:36.891423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:36.892320 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:36.892330 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:36.893058 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:36.893062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:36.893174 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:36.893720 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:36.893736 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:36.896278 (+ 2542us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":285,"threads_started":1}
W20260929 21:00:36.896670 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:36.888602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34697 (local address 127.0.97.62:39497)
0929 21:00:36.888845 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:36.888855 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:36.889216 (+ 361us) server_negotiation.cc:407] Connection header received
0929 21:00:36.889359 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:36.889362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:36.889421 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:36.889500 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:36.890402 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:36.891303 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:36.892445 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:36.892906 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:36.893020 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:36.896458 (+ 3438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:34697: BlockingRecv error: recv got EOF from 127.0.97.2:34697 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 21:00:37 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:37 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:37.343377 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:37.335230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.3:50785)
0929 21:00:37.335706 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:37.335727 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:37.335885 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:37.336207 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:37.336212 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:37.336619 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:37.336890 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:37.336904 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:37.337962 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:37.337968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:37.338894 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:37.338904 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:37.339620 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:37.339626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:37.339792 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:37.340354 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:37.340371 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:37.343150 (+ 2779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":254,"threads_started":1}
W20260929 21:00:37.343621 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:37.335279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50785 (local address 127.0.97.62:39497)
0929 21:00:37.335526 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:37.335533 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:37.335830 (+ 297us) server_negotiation.cc:407] Connection header received
0929 21:00:37.335997 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:37.336001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:37.336059 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:37.336175 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:37.337063 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:37.337799 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:37.339055 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:37.339458 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:37.339584 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:37.343387 (+ 3803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:50785: BlockingRecv error: recv got EOF from 127.0.97.3:50785 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 21:00:37 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:37 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:37.477946 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:37.470334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.1:33229)
0929 21:00:37.470784 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:37.470800 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:37.470915 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:37.471197 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:37.471200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:37.471554 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:37.471686 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:37.471694 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:37.472725 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:37.472732 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:37.473664 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:37.473675 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:37.474376 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:37.474382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:37.474557 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:37.475089 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:37.475107 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:37.477724 (+ 2617us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":264,"threads_started":1}
W20260929 21:00:37.478164 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:37.470389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33229 (local address 127.0.97.62:39497)
0929 21:00:37.470622 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:37.470628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:37.470875 (+ 247us) server_negotiation.cc:407] Connection header received
0929 21:00:37.471007 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:37.471010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:37.471080 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:37.471161 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:37.471849 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:37.472586 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:37.473804 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:37.474220 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:37.474356 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:37.477902 (+ 3546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:33229: BlockingRecv error: recv got EOF from 127.0.97.1:33229 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 21:00:37 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:37 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:37.905417 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:37.897525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.2:45891)
0929 21:00:37.898033 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:37.898050 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:37.898154 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:37.898442 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:37.898445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:37.898799 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:37.898944 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:37.898952 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:37.899854 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:37.899858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:37.900970 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:37.900982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:37.901713 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:37.901718 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:37.901921 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:37.902508 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:37.902525 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:37.905192 (+ 2667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":298,"threads_started":1}
W20260929 21:00:37.905596 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:37.897583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45891 (local address 127.0.97.62:39497)
0929 21:00:37.897747 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:37.897751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:37.898122 (+ 371us) server_negotiation.cc:407] Connection header received
0929 21:00:37.898247 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:37.898251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:37.898306 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:37.898394 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:37.899063 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:37.899734 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:37.901114 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:37.901549 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:37.901684 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:37.905358 (+ 3674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:45891: BlockingRecv error: recv got EOF from 127.0.97.2:45891 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 21:00:38 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:38 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:38.352697 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:38.344542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.3:44323)
0929 21:00:38.345053 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:38.345072 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:38.345216 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:38.345567 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:38.345572 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:38.345951 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:38.346137 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:38.346155 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:38.347157 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:38.347161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:38.348073 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:38.348083 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:38.348698 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:38.348702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:38.348812 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:38.349484 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:38.349501 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:38.352341 (+ 2840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":283,"threads_started":1}
W20260929 21:00:38.352864 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:38.344687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44323 (local address 127.0.97.62:39497)
0929 21:00:38.344922 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:38.344928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:38.345151 (+ 223us) server_negotiation.cc:407] Connection header received
0929 21:00:38.345344 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:38.345349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:38.345431 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:38.345535 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:38.346310 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:38.347039 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:38.348191 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:38.348576 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:38.348665 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:38.352575 (+ 3910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:44323: BlockingRecv error: recv got EOF from 127.0.97.3:44323 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 29 21:00:38 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:38 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:38.487296 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:38.478841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.1:33001)
0929 21:00:38.479342 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:38.479361 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:38.479472 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:38.479782 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:38.479785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:38.480171 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:38.480336 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:38.480343 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:38.481416 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:38.481420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:38.482361 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:38.482377 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:38.483441 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:38.483448 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:38.483605 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:38.484238 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:38.484256 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:38.487062 (+ 2806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":277,"threads_started":1}
W20260929 21:00:38.487527 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:38.478922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33001 (local address 127.0.97.62:39497)
0929 21:00:38.479112 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:38.479118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:38.479441 (+ 323us) server_negotiation.cc:407] Connection header received
0929 21:00:38.479570 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:38.479574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:38.479655 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:38.479750 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:38.480486 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:38.481290 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:38.482548 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:38.483255 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:38.483392 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:38.487301 (+ 3909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:33001: BlockingRecv error: recv got EOF from 127.0.97.1:33001 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 21:00:38 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:38 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:38.914557 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:38.906257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.2:37905)
0929 21:00:38.906782 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:38.906799 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:38.906909 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:38.907305 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:38.907308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:38.907697 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:38.907859 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:38.907866 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:38.908779 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:38.908782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:38.909694 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:38.909705 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:38.910408 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:38.910412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:38.910555 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:38.911286 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:38.911310 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:38.914323 (+ 3013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":292,"threads_started":1}
W20260929 21:00:38.914749 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:38.906339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37905 (local address 127.0.97.62:39497)
0929 21:00:38.906518 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:38.906522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:38.906885 (+ 363us) server_negotiation.cc:407] Connection header received
0929 21:00:38.907115 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:38.907119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:38.907180 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:38.907269 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:38.907979 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:38.908664 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:38.909811 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:38.910255 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:38.910372 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:38.914502 (+ 4130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37905: BlockingRecv error: recv got EOF from 127.0.97.2:37905 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 29 21:00:39 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:39 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:39.367805 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:39.353710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.3:37559)
0929 21:00:39.354192 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:39.354211 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:39.354358 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:39.354691 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:39.354696 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:39.355123 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:39.355289 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:39.355298 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:39.356363 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:39.356370 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:39.357382 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:39.357394 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:39.358088 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:39.358094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:39.358268 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:39.364728 (+ 6460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:39.364746 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:39.367572 (+ 2826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":261,"threads_started":1}
W20260929 21:00:39.367986 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:39.353795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37559 (local address 127.0.97.62:39497)
0929 21:00:39.354036 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:39.354043 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:39.354289 (+ 246us) server_negotiation.cc:407] Connection header received
0929 21:00:39.354493 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:39.354498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:39.354564 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:39.354664 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:39.355440 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:39.356235 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:39.357535 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:39.357937 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:39.358081 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:39.367775 (+ 9694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:37559: BlockingRecv error: recv got EOF from 127.0.97.3:37559 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 29 21:00:39 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:39 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:39.496516 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:39.488267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.1:41589)
0929 21:00:39.488760 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:39.488776 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:39.488962 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:39.489306 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:39.489309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:39.489671 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:39.489823 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:39.489830 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:39.490902 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:39.490906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:39.491809 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:39.491820 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:39.492629 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:39.492635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:39.492774 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:39.493564 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:39.493596 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:39.496266 (+ 2670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":282,"threads_started":1}
W20260929 21:00:39.496754 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:39.488310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41589 (local address 127.0.97.62:39497)
0929 21:00:39.488553 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:39.488562 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:39.488900 (+ 338us) server_negotiation.cc:407] Connection header received
0929 21:00:39.489076 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:39.489080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:39.489166 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:39.489266 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:39.489950 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:39.490773 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:39.491956 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:39.492435 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:39.492533 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:39.496474 (+ 3941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:41589: BlockingRecv error: recv got EOF from 127.0.97.1:41589 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 21:00:39 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:39 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:39.923437 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:39.915524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.2:56767)
0929 21:00:39.916018 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:39.916035 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:39.916154 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:39.916440 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:39.916443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:39.916790 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:39.916967 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:39.916974 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:39.917871 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:39.917874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:39.918760 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:39.918770 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:39.919384 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:39.919387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:39.919510 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:39.920187 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:39.920205 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:39.923198 (+ 2993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":259,"threads_started":1}
W20260929 21:00:39.923593 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:39.915525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56767 (local address 127.0.97.62:39497)
0929 21:00:39.915731 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:39.915740 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:39.916118 (+ 378us) server_negotiation.cc:407] Connection header received
0929 21:00:39.916254 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:39.916258 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:39.916316 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:39.916397 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:39.917079 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:39.917754 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:39.918875 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:39.919259 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:39.919346 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:39.923372 (+ 4026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56767: BlockingRecv error: recv got EOF from 127.0.97.2:56767 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 21:00:40 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:40 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:40.376817 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:40.368744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.3:40803)
0929 21:00:40.369302 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:40.369322 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:40.369475 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:40.369804 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:40.369809 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:40.370197 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:40.370342 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:40.370359 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:40.371370 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:40.371376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:40.372283 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:40.372293 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:40.373008 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:40.373014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:40.373199 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:40.373788 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:40.373804 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:40.376481 (+ 2677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":307,"threads_started":1}
W20260929 21:00:40.376981 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:40.368858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40803 (local address 127.0.97.62:39497)
0929 21:00:40.369039 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:40.369047 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:40.369416 (+ 369us) server_negotiation.cc:407] Connection header received
0929 21:00:40.369602 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:40.369608 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:40.369673 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:40.369772 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:40.370490 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:40.371226 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:40.372426 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:40.372814 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:40.373002 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:40.376685 (+ 3683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:40803: BlockingRecv error: recv got EOF from 127.0.97.3:40803 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 21:00:40 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:40 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:40.507468 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:40.497559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.1:58881)
0929 21:00:40.498062 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:40.498082 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:40.498247 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:40.498622 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:40.498629 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:40.499034 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:40.499254 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:40.499270 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:40.500273 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:40.500279 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:40.501651 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:40.501668 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:40.502844 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:40.502854 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:40.503069 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:40.503762 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:40.503781 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:40.507062 (+ 3281us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":345,"threads_started":1}
W20260929 21:00:40.507591 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:40.497668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58881 (local address 127.0.97.62:39497)
0929 21:00:40.497903 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:40.497910 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:40.498196 (+ 286us) server_negotiation.cc:407] Connection header received
0929 21:00:40.498378 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:40.498383 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:40.498475 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:40.498581 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:40.499428 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:40.500119 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:40.501830 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:40.502607 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:40.502782 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:40.507303 (+ 4521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58881: BlockingRecv error: recv got EOF from 127.0.97.1:58881 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 29 21:00:40 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:40 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:40.932431 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:40.924412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.2:59757)
0929 21:00:40.924974 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:40.924990 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:40.925100 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:40.925393 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:40.925396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:40.925795 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:40.925949 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:40.925956 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:40.926929 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:40.926934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:40.927836 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:40.927847 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:40.928554 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:40.928559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:40.928672 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:40.929299 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:40.929316 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:40.932205 (+ 2889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":279,"threads_started":1}
W20260929 21:00:40.932608 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:40.924649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59757 (local address 127.0.97.62:39497)
0929 21:00:40.924914 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:40.924919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:40.925148 (+ 229us) server_negotiation.cc:407] Connection header received
0929 21:00:40.925197 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:40.925201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:40.925257 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:40.925350 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:40.926070 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:40.926811 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:40.927961 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:40.928407 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:40.928498 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:40.932383 (+ 3885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:59757: BlockingRecv error: recv got EOF from 127.0.97.2:59757 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 21:00:41 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:41 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:41.386319 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:41.377843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.3:55661)
0929 21:00:41.378387 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:41.378406 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:41.378516 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:41.378764 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:41.378767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:41.379270 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:41.379488 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:41.379500 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:41.380906 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:41.380912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:41.381819 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:41.381829 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:41.382576 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:41.382582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:41.382697 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:41.383269 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:41.383285 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:41.386095 (+ 2810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":291,"threads_started":1}
W20260929 21:00:41.386472 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:41.377905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55661 (local address 127.0.97.62:39497)
0929 21:00:41.378073 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:41.378077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:41.378479 (+ 402us) server_negotiation.cc:407] Connection header received
0929 21:00:41.378609 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:41.378613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:41.378672 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:41.378750 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:41.379653 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:41.380721 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:41.381946 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:41.382328 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:41.382417 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:41.386246 (+ 3829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:55661: BlockingRecv error: recv got EOF from 127.0.97.3:55661 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 21:00:41 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:41 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:41.516633 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:41.508812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.1:37315)
0929 21:00:41.509296 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:41.509314 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:41.509434 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:41.509743 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:41.509746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:41.510095 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:41.510264 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:41.510272 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:41.511181 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:41.511185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:41.512105 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:41.512116 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:41.512774 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:41.512778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:41.512980 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:41.513567 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:41.513583 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:41.516273 (+ 2690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":264,"threads_started":1}
W20260929 21:00:41.516706 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:41.508905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37315 (local address 127.0.97.62:39497)
0929 21:00:41.509132 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:41.509137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:41.509397 (+ 260us) server_negotiation.cc:407] Connection header received
0929 21:00:41.509526 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:41.509530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:41.509620 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:41.509701 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:41.510380 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:41.511058 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:41.512221 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:41.512658 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:41.512748 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:41.516476 (+ 3728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37315: BlockingRecv error: recv got EOF from 127.0.97.1:37315 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Sep 29 21:00:41 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:41 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:41.940740 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:41.933365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.2:47989)
0929 21:00:41.933805 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:41.933821 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:41.933930 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:41.934200 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:41.934203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:41.934546 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:41.934707 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:41.934715 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:41.935637 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:41.935640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:41.936534 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:41.936544 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:41.937144 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:41.937148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:41.937264 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:41.937834 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:41.937852 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:41.940501 (+ 2649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":227,"threads_started":1}
W20260929 21:00:41.940909 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:41.933421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47989 (local address 127.0.97.62:39497)
0929 21:00:41.933641 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:41.933647 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:41.933900 (+ 253us) server_negotiation.cc:407] Connection header received
0929 21:00:41.934046 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:41.934050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:41.934110 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:41.934189 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:41.934830 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:41.935520 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:41.936645 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:41.937036 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:41.937122 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:41.940672 (+ 3550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:47989: BlockingRecv error: recv got EOF from 127.0.97.2:47989 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 21:00:42 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:42 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:42.395787 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:42.387297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.3:51417)
0929 21:00:42.387750 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:42.387771 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:42.387936 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:42.388283 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:42.388288 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:42.388676 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:42.388904 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:42.388915 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:42.389936 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:42.389942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:42.390916 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:42.390927 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:42.391630 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:42.391635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:42.391829 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:42.392543 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:42.392584 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:42.395539 (+ 2955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":240,"threads_started":1}
W20260929 21:00:42.395973 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:42.387373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51417 (local address 127.0.97.62:39497)
0929 21:00:42.387591 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:42.387598 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:42.387859 (+ 261us) server_negotiation.cc:407] Connection header received
0929 21:00:42.388069 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:42.388076 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:42.388152 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:42.388254 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:42.389051 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:42.389785 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:42.391052 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:42.391486 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:42.391622 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:42.395730 (+ 4108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:51417: BlockingRecv error: recv got EOF from 127.0.97.3:51417 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 29 21:00:42 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:42 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:42.525671 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:42.517791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.1:36325)
0929 21:00:42.518261 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:42.518277 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:42.518385 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:42.518688 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:42.518691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:42.519066 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:42.519253 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:42.519261 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:42.520142 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:42.520145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:42.521225 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:42.521241 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:42.522083 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:42.522089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:42.522258 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:42.522805 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:42.522821 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:42.525426 (+ 2605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":272,"threads_started":1}
W20260929 21:00:42.525784 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:42.517841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36325 (local address 127.0.97.62:39497)
0929 21:00:42.518046 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:42.518056 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:42.518353 (+ 297us) server_negotiation.cc:407] Connection header received
0929 21:00:42.518477 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:42.518481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:42.518572 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:42.518652 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:42.519372 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:42.520033 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:42.521360 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:42.521927 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:42.522063 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:42.525598 (+ 3535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:36325: BlockingRecv error: recv got EOF from 127.0.97.1:36325 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 21:00:42 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 21:00:42 dist-test-slave-1dms krb5kdc[8619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 21:00:42.956214 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:42.941779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39497 (local address 127.0.97.2:58879)
0929 21:00:42.942266 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:42.942282 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:42.942443 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:42.942779 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:42.942783 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:42.943198 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:42.943429 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:42.943440 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:42.944807 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:42.944814 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:42.945921 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:42.945933 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:42.946739 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:42.946743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:42.946857 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:42.953050 (+ 6193us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:42.953068 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:42.955828 (+ 2760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39497: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":281,"threads_started":1}
W20260929 21:00:42.956300 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:42.941855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58879 (local address 127.0.97.62:39497)
0929 21:00:42.942087 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:42.942094 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:42.942362 (+ 268us) server_negotiation.cc:407] Connection header received
0929 21:00:42.942575 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:42.942583 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:42.942656 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:42.942754 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:42.943563 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:42.944600 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:42.946079 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:42.946607 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:42.946702 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:42.956050 (+ 9348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58879: BlockingRecv error: recv got EOF from 127.0.97.2:58879 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 21:00:43.219166 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 9206
I20260929 21:00:43.230762 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:43.220181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54320 (local address 127.0.97.1:43645)
0929 21:00:43.220712 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:43.220719 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:43.220746 (+ 27us) server_negotiation.cc:407] Connection header received
0929 21:00:43.220800 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:43.220806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:43.221101 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:43.221260 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:43.222555 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:43.223525 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:43.224687 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:43.225283 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:43.225420 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:43.227693 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:43.227700 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:43.227714 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:43.227743 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:43.230247 (+ 2504us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 21:00:43.230488 (+ 241us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54320: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":272,"threads_started":1}
W20260929 21:00:43.230872 9666 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54320: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 21:00:43.230907 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 9206: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:43645: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 21:00:43.231125 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 9206
I20260929 21:00:43.247704 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 9357
I20260929 21:00:43.259581 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:43.248763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48896 (local address 127.0.97.2:44387)
0929 21:00:43.249418 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:43.249429 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:43.249468 (+ 39us) server_negotiation.cc:407] Connection header received
0929 21:00:43.249530 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:43.249536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:43.249784 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:43.250006 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:43.251363 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:43.252396 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:43.253592 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:43.254124 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:43.254218 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:43.256496 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:43.256504 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:43.256517 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:43.256550 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:43.259072 (+ 2522us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 21:00:43.259304 (+ 232us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48896: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":353,"threads_started":1}
W20260929 21:00:43.259677 9667 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48896: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 21:00:43.259703 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 9357: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:44387: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 21:00:43.259868 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 9357
I20260929 21:00:43.276237 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 9488
I20260929 21:00:43.287278 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:43.277170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56662 (local address 127.0.97.3:42169)
0929 21:00:43.277767 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:43.277774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:43.277800 (+ 26us) server_negotiation.cc:407] Connection header received
0929 21:00:43.277861 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:43.277870 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:43.278094 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:43.278246 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:43.279441 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:43.280406 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:43.281578 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:43.282110 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:43.282207 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:43.284509 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:43.284517 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:43.284530 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:43.284567 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:43.286820 (+ 2253us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 21:00:43.287051 (+ 231us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56662: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":309,"threads_started":1}
W20260929 21:00:43.287415 9668 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56662: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 21:00:43.287438 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 9488: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:42169: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 21:00:43.287590 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 9488
I20260929 21:00:43.303982 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 9137
I20260929 21:00:43.305351 9199 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:00:43.418941 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 9137
2026-09-29T21:00:43Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20924 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T21:00:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:00:43Z Disabled control of system clock
I20260929 21:00:43.492607 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35933
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:43987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35933
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 21:00:43.807265 9673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:43.807541 9673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:43.807683 9673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:43.814711 9673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:00:43.814807 9673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:43.814877 9673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:43.814965 9673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:00:43.815047 9673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:00:43.823805 9673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35933
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:00:43.825048 9673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:43.826784 9673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:43.838356 9678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:43.838403 9679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:43.838433 9681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:43.838953 9673 server_base.cc:1034] running on GCE node
I20260929 21:00:43.839665 9673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:43.841048 9673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:43.842249 9673 hybrid_clock.cc:648] HybridClock initialized: now 1790715643842210 us; error 28 us; skew 500 ppm
I20260929 21:00:43.842598 9673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 21:00:43.844740 9673 webserver.cc:460] Webserver started at http://127.0.97.62:38551/ using document root <none> and password file <none>
I20260929 21:00:43.845463 9673 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:43.845577 9673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:43.845928 9673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:00:43.848636 9673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "f851feab182246cca49c2613f2095ba7"
format_stamp: "Formatted at 2026-09-29 21:00:43 on dist-test-slave-1dms"
server_key: "53709379007bf3aa04a2fca92a514779"
server_key_iv: "447dcff4f1af2fd8343b8aa03c921229"
server_key_version: "encryptionkey@0"
I20260929 21:00:43.849469 9673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "f851feab182246cca49c2613f2095ba7"
format_stamp: "Formatted at 2026-09-29 21:00:43 on dist-test-slave-1dms"
server_key: "53709379007bf3aa04a2fca92a514779"
server_key_iv: "447dcff4f1af2fd8343b8aa03c921229"
server_key_version: "encryptionkey@0"
I20260929 21:00:43.854233 9673 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 21:00:43.857331 9687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:43.858558 9673 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260929 21:00:43.858734 9673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "f851feab182246cca49c2613f2095ba7"
format_stamp: "Formatted at 2026-09-29 21:00:43 on dist-test-slave-1dms"
server_key: "53709379007bf3aa04a2fca92a514779"
server_key_iv: "447dcff4f1af2fd8343b8aa03c921229"
server_key_version: "encryptionkey@0"
I20260929 21:00:43.858950 9673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:43.878732 9673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:43.879535 9673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:00:43.879809 9673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:43.897536 9673 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35933
I20260929 21:00:43.897557 9738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35933 every 8 connection(s)
I20260929 21:00:43.898808 9673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:00:43.900985 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 9673
I20260929 21:00:43.901180 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:00:43.901783 388 external_mini_cluster.cc:1428] Setting key 795ab9532a51d9802e88d683007b6d53
I20260929 21:00:43.902757 9739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:00:43.917131 9742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:43.903760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39560 (local address 127.0.97.62:35933)
0929 21:00:43.904774 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:43.904790 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:43.904868 (+ 78us) server_negotiation.cc:407] Connection header received
0929 21:00:43.907350 (+ 2482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:43.907379 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:43.907987 (+ 608us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:43.908440 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:43.908993 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:43.910924 (+ 1931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:43.912134 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:43.913166 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:43.913355 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:43.914437 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:43.914448 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:43.914471 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:00:43.915201 (+ 730us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:43.915301 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:00:43.915328 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:43.915444 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:43.915598 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 21:00:43.915901 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":314,"threads_started":1}
I20260929 21:00:43.917539 9739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f851feab182246cca49c2613f2095ba7: Bootstrap starting.
I20260929 21:00:43.920588 9739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f851feab182246cca49c2613f2095ba7: Neither blocks nor log segments found. Creating new log.
I20260929 21:00:43.921595 9739 log.cc:824] T 00000000000000000000000000000000 P f851feab182246cca49c2613f2095ba7: Log is configured to *not* fsync() on all Append() calls
I20260929 21:00:43.924063 9739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f851feab182246cca49c2613f2095ba7: No bootstrap required, opened a new log
I20260929 21:00:43.931392 9739 raft_consensus.cc:348] T 00000000000000000000000000000000 P f851feab182246cca49c2613f2095ba7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f851feab182246cca49c2613f2095ba7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35933 } }
I20260929 21:00:43.931721 9739 raft_consensus.cc:374] T 00000000000000000000000000000000 P f851feab182246cca49c2613f2095ba7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:00:43.931794 9739 raft_consensus.cc:729] T 00000000000000000000000000000000 P f851feab182246cca49c2613f2095ba7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f851feab182246cca49c2613f2095ba7, State: Initialized, Role: FOLLOWER
I20260929 21:00:43.932257 9739 consensus_queue.cc:260] T 00000000000000000000000000000000 P f851feab182246cca49c2613f2095ba7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f851feab182246cca49c2613f2095ba7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35933 } }
I20260929 21:00:43.932440 9739 raft_consensus.cc:388] T 00000000000000000000000000000000 P f851feab182246cca49c2613f2095ba7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:00:43.932531 9739 raft_consensus.cc:482] T 00000000000000000000000000000000 P f851feab182246cca49c2613f2095ba7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:00:43.932648 9739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f851feab182246cca49c2613f2095ba7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:00:43.934662 9739 raft_consensus.cc:504] T 00000000000000000000000000000000 P f851feab182246cca49c2613f2095ba7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f851feab182246cca49c2613f2095ba7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35933 } }
I20260929 21:00:43.935070 9739 leader_election.cc:302] T 00000000000000000000000000000000 P f851feab182246cca49c2613f2095ba7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f851feab182246cca49c2613f2095ba7; no voters:
I20260929 21:00:43.935577 9739 leader_election.cc:288] T 00000000000000000000000000000000 P f851feab182246cca49c2613f2095ba7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 21:00:43.935724 9744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f851feab182246cca49c2613f2095ba7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:00:43.936383 9744 raft_consensus.cc:686] T 00000000000000000000000000000000 P f851feab182246cca49c2613f2095ba7 [term 1 LEADER]: Becoming Leader. State: Replica: f851feab182246cca49c2613f2095ba7, State: Running, Role: LEADER
I20260929 21:00:43.936918 9744 consensus_queue.cc:237] T 00000000000000000000000000000000 P f851feab182246cca49c2613f2095ba7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f851feab182246cca49c2613f2095ba7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35933 } }
I20260929 21:00:43.937630 9739 sys_catalog.cc:564] T 00000000000000000000000000000000 P f851feab182246cca49c2613f2095ba7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:00:43.939519 9746 sys_catalog.cc:455] T 00000000000000000000000000000000 P f851feab182246cca49c2613f2095ba7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f851feab182246cca49c2613f2095ba7. Latest consensus state: current_term: 1 leader_uuid: "f851feab182246cca49c2613f2095ba7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f851feab182246cca49c2613f2095ba7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35933 } } }
I20260929 21:00:43.939913 9746 sys_catalog.cc:458] T 00000000000000000000000000000000 P f851feab182246cca49c2613f2095ba7 [sys.catalog]: This master's current role is: LEADER
I20260929 21:00:43.939666 9745 sys_catalog.cc:455] T 00000000000000000000000000000000 P f851feab182246cca49c2613f2095ba7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f851feab182246cca49c2613f2095ba7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f851feab182246cca49c2613f2095ba7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35933 } } }
I20260929 21:00:43.941583 9745 sys_catalog.cc:458] T 00000000000000000000000000000000 P f851feab182246cca49c2613f2095ba7 [sys.catalog]: This master's current role is: LEADER
I20260929 21:00:43.941648 9752 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:00:43.946820 9752 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:00:43.955437 9752 catalog_manager.cc:1347] Generated new cluster ID: 803f0622d7914d4f8e1af4591ff2bc7b
I20260929 21:00:43.955572 9752 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:00:43.963661 9752 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:00:43.965370 9752 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:00:43.977648 9752 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f851feab182246cca49c2613f2095ba7: Generated new TSK 0
I20260929 21:00:43.978677 9752 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 21:00:43.989303 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35933
--builtin_ntp_servers=127.0.97.20:43987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 21:00:44.315163 9763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:44.315425 9763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:44.315569 9763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:44.322506 9763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:44.322615 9763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:44.322781 9763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 21:00:44.331344 9763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:00:44.332566 9763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:44.334357 9763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:44.347761 9768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:44.347815 9771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:44.347842 9769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:44.348961 9763 server_base.cc:1034] running on GCE node
I20260929 21:00:44.349598 9763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:44.350607 9763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:44.351893 9763 hybrid_clock.cc:648] HybridClock initialized: now 1790715644351841 us; error 66 us; skew 500 ppm
I20260929 21:00:44.352236 9763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 21:00:44.354581 9763 webserver.cc:460] Webserver started at http://127.0.97.1:35385/ using document root <none> and password file <none>
I20260929 21:00:44.355314 9763 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:44.355454 9763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:44.355808 9763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:00:44.359026 9763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-0/data/instance:
uuid: "f7e6995973494e3d935a761525cc1d1c"
format_stamp: "Formatted at 2026-09-29 21:00:44 on dist-test-slave-1dms"
server_key: "408e441805f97ef742cfa560b4c844fb"
server_key_iv: "41983b4eb83aaa9ffd58aab7f7ab24ee"
server_key_version: "encryptionkey@0"
I20260929 21:00:44.359896 9763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f7e6995973494e3d935a761525cc1d1c"
format_stamp: "Formatted at 2026-09-29 21:00:44 on dist-test-slave-1dms"
server_key: "408e441805f97ef742cfa560b4c844fb"
server_key_iv: "41983b4eb83aaa9ffd58aab7f7ab24ee"
server_key_version: "encryptionkey@0"
I20260929 21:00:44.365144 9763 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 21:00:44.368458 9777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:44.369855 9763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:00:44.370025 9763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-0/wal
uuid: "f7e6995973494e3d935a761525cc1d1c"
format_stamp: "Formatted at 2026-09-29 21:00:44 on dist-test-slave-1dms"
server_key: "408e441805f97ef742cfa560b4c844fb"
server_key_iv: "41983b4eb83aaa9ffd58aab7f7ab24ee"
server_key_version: "encryptionkey@0"
I20260929 21:00:44.370234 9763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:44.398234 9763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:44.399138 9763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:00:44.399437 9763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:44.400398 9763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:00:44.402151 9763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:00:44.402274 9763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:44.402403 9763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:00:44.402496 9763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:44.435725 9763 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44115
I20260929 21:00:44.435837 9889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44115 every 8 connection(s)
I20260929 21:00:44.437069 9763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
I20260929 21:00:44.443117 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 9763
I20260929 21:00:44.443300 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-0/wal/instance
I20260929 21:00:44.444103 388 external_mini_cluster.cc:1428] Setting key 6aa46e322fd354dd68e58f4a9ee26ed1
I20260929 21:00:44.446921 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35933
--builtin_ntp_servers=127.0.97.20:43987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 21:00:44.450269 9892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:44.439709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35303 (local address 127.0.97.62:35933)
0929 21:00:44.440137 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:44.440141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:44.441321 (+ 1180us) server_negotiation.cc:407] Connection header received
0929 21:00:44.443054 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:44.443060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:44.443155 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:44.443285 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:44.444823 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:44.445789 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:44.447447 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:44.447848 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:44.447980 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:44.449439 (+ 1459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:44.449444 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:44.449447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:00:44.449464 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:44.449522 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:00:44.449536 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:44.449642 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:44.449859 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 21:00:44.449951 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":241,"threads_started":1}
I20260929 21:00:44.451275 9893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:44.440186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35933 (local address 127.0.97.1:35303)
0929 21:00:44.441133 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:44.441170 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:44.442799 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:44.443624 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:44.443638 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:44.443766 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:44.444619 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:44.444645 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:44.445948 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:44.445954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:44.447308 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:44.447320 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:44.448069 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:44.448077 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:44.448324 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:44.449072 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 21:00:44.449074 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:44.449113 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 21:00:44.449222 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:44.449642 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:44.449655 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:44.449666 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 21:00:44.449851 (+ 185us) client_negotiation.cc:240] Negotiation successful
0929 21:00:44.450264 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":269,"threads_started":1}
I20260929 21:00:44.452796 9890 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35933
I20260929 21:00:44.453140 9890 heartbeater.cc:464] Registering TS with master...
I20260929 21:00:44.453853 9890 heartbeater.cc:511] Master 127.0.97.62:35933 requested a full tablet report, sending...
I20260929 21:00:44.455956 9704 ts_manager.cc:194] Registered new tserver with Master: f7e6995973494e3d935a761525cc1d1c (127.0.97.1:44115)
I20260929 21:00:44.458123 9704 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:35303
W20260929 21:00:44.773876 9894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:44.774133 9894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:44.774276 9894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:44.781361 9894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:44.781448 9894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:44.781544 9894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 21:00:44.790160 9894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:00:44.791312 9894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:44.792984 9894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:44.805343 9899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:44.805425 9900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:44.805477 9902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:44.806406 9894 server_base.cc:1034] running on GCE node
I20260929 21:00:44.807020 9894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:44.807842 9894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:44.809046 9894 hybrid_clock.cc:648] HybridClock initialized: now 1790715644809007 us; error 50 us; skew 500 ppm
I20260929 21:00:44.809347 9894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 21:00:44.811458 9894 webserver.cc:460] Webserver started at http://127.0.97.2:39029/ using document root <none> and password file <none>
I20260929 21:00:44.812106 9894 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:44.812239 9894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:44.812557 9894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:00:44.815279 9894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-1/data/instance:
uuid: "4e3093f6901b47f18785ae4abffc370e"
format_stamp: "Formatted at 2026-09-29 21:00:44 on dist-test-slave-1dms"
server_key: "1538407827456ba21116b7cfab89966f"
server_key_iv: "985ef03b990d9fbeb9953e637e22068b"
server_key_version: "encryptionkey@0"
I20260929 21:00:44.816042 9894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4e3093f6901b47f18785ae4abffc370e"
format_stamp: "Formatted at 2026-09-29 21:00:44 on dist-test-slave-1dms"
server_key: "1538407827456ba21116b7cfab89966f"
server_key_iv: "985ef03b990d9fbeb9953e637e22068b"
server_key_version: "encryptionkey@0"
I20260929 21:00:44.820822 9894 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260929 21:00:44.824002 9908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:44.825224 9894 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.003s
I20260929 21:00:44.825397 9894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-1/wal
uuid: "4e3093f6901b47f18785ae4abffc370e"
format_stamp: "Formatted at 2026-09-29 21:00:44 on dist-test-slave-1dms"
server_key: "1538407827456ba21116b7cfab89966f"
server_key_iv: "985ef03b990d9fbeb9953e637e22068b"
server_key_version: "encryptionkey@0"
I20260929 21:00:44.825606 9894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:44.853152 9894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:44.853986 9894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:00:44.854254 9894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:44.855163 9894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:00:44.856566 9894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:00:44.856685 9894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:44.856823 9894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:00:44.856935 9894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260929 21:00:44.889700 9894 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41065
I20260929 21:00:44.889731 10020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41065 every 8 connection(s)
I20260929 21:00:44.890964 9894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
I20260929 21:00:44.900085 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 9894
I20260929 21:00:44.900241 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-1/wal/instance
I20260929 21:00:44.900772 388 external_mini_cluster.cc:1428] Setting key 3f126a520d6f41883b3c9de581a3bc45
I20260929 21:00:44.903180 9892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:44.893428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49925 (local address 127.0.97.62:35933)
0929 21:00:44.893595 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:44.893599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:44.894906 (+ 1307us) server_negotiation.cc:407] Connection header received
0929 21:00:44.896624 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:44.896629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:44.896720 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:44.896829 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:44.897906 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:44.898683 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:44.900298 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:44.900708 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:44.900866 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:44.902281 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:44.902285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:44.902289 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:00:44.902305 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:44.902333 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:00:44.902345 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:44.902450 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:44.902779 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 21:00:44.902870 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 21:00:44.903889 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35933
--builtin_ntp_servers=127.0.97.20:43987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 21:00:44.904352 10023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:44.893885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35933 (local address 127.0.97.2:49925)
0929 21:00:44.894752 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:44.894791 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:44.896358 (+ 1567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:44.897115 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:44.897124 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:44.897215 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:44.897745 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:44.897761 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:44.898832 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:44.898840 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:44.900077 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:44.900088 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:44.901002 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:44.901016 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:44.901207 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:44.901960 (+ 753us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 21:00:44.901964 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:44.902020 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 21:00:44.902092 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:44.902541 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:44.902553 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:44.902561 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 21:00:44.902779 (+ 218us) client_negotiation.cc:240] Negotiation successful
0929 21:00:44.903163 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":269,"threads_started":1}
I20260929 21:00:44.905761 10021 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35933
I20260929 21:00:44.906071 10021 heartbeater.cc:464] Registering TS with master...
I20260929 21:00:44.906812 10021 heartbeater.cc:511] Master 127.0.97.62:35933 requested a full tablet report, sending...
I20260929 21:00:44.908121 9704 ts_manager.cc:194] Registered new tserver with Master: 4e3093f6901b47f18785ae4abffc370e (127.0.97.2:41065)
I20260929 21:00:44.909080 9704 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:49925
W20260929 21:00:45.229981 10024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:45.230297 10024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:45.230464 10024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:45.237932 10024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:45.238073 10024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:45.238258 10024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 21:00:45.247601 10024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:00:45.248919 10024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:45.250712 10024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:45.263747 10029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:45.263770 10032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:45.264382 10030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:45.264393 10024 server_base.cc:1034] running on GCE node
I20260929 21:00:45.265095 10024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:45.266129 10024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:45.267380 10024 hybrid_clock.cc:648] HybridClock initialized: now 1790715645267333 us; error 62 us; skew 500 ppm
I20260929 21:00:45.267722 10024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 21:00:45.270032 10024 webserver.cc:460] Webserver started at http://127.0.97.3:39529/ using document root <none> and password file <none>
I20260929 21:00:45.270709 10024 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:45.270844 10024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:45.271169 10024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:00:45.274125 10024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-2/data/instance:
uuid: "a48cdf649e95442ebd9cfcf8c1a1c86e"
format_stamp: "Formatted at 2026-09-29 21:00:45 on dist-test-slave-1dms"
server_key: "68c38d00a1b1b50b72ce064610837cd5"
server_key_iv: "37169a53e8b0c02b932831bd25e5460e"
server_key_version: "encryptionkey@0"
I20260929 21:00:45.275166 10024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a48cdf649e95442ebd9cfcf8c1a1c86e"
format_stamp: "Formatted at 2026-09-29 21:00:45 on dist-test-slave-1dms"
server_key: "68c38d00a1b1b50b72ce064610837cd5"
server_key_iv: "37169a53e8b0c02b932831bd25e5460e"
server_key_version: "encryptionkey@0"
I20260929 21:00:45.282457 10024 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s
I20260929 21:00:45.285936 10038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:45.287376 10024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 21:00:45.287576 10024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-2/wal
uuid: "a48cdf649e95442ebd9cfcf8c1a1c86e"
format_stamp: "Formatted at 2026-09-29 21:00:45 on dist-test-slave-1dms"
server_key: "68c38d00a1b1b50b72ce064610837cd5"
server_key_iv: "37169a53e8b0c02b932831bd25e5460e"
server_key_version: "encryptionkey@0"
I20260929 21:00:45.287806 10024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:45.305421 10024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:45.306299 10024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:00:45.306584 10024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:45.307514 10024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:00:45.309000 10024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:00:45.309115 10024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:45.309252 10024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:00:45.309337 10024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:45.341813 10024 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44285
I20260929 21:00:45.341868 10150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44285 every 8 connection(s)
I20260929 21:00:45.343114 10024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
I20260929 21:00:45.347216 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 10024
I20260929 21:00:45.347371 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790715556572573-388-0/minicluster-data/ts-2/wal/instance
I20260929 21:00:45.347961 388 external_mini_cluster.cc:1428] Setting key 42e9a72a8b9b9f2158e42c6c3aa956ff
I20260929 21:00:45.355893 9892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:45.345717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50291 (local address 127.0.97.62:35933)
0929 21:00:45.345894 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:45.345899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:45.347569 (+ 1670us) server_negotiation.cc:407] Connection header received
0929 21:00:45.349994 (+ 2425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:45.350000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:45.350089 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:45.350203 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:45.351380 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:45.352106 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:45.353625 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:45.353997 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:45.354088 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:45.355308 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:45.355312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:45.355315 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:00:45.355331 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:45.355355 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:00:45.355365 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:45.355430 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:45.355612 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 21:00:45.355675 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 21:00:45.356935 10153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:45.346285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35933 (local address 127.0.97.3:50291)
0929 21:00:45.347391 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:45.347453 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:45.349638 (+ 2185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:45.350400 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:45.350408 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:45.350494 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:45.351236 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:45.351254 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:45.352226 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:45.352230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:45.353505 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:45.353520 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:45.354165 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:45.354169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:45.354361 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:45.355052 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 21:00:45.355055 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:45.355092 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 21:00:45.355160 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:45.355445 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:45.355452 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:45.355457 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 21:00:45.355590 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 21:00:45.355921 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":272,"threads_started":1}
I20260929 21:00:45.358124 10151 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35933
I20260929 21:00:45.358398 10151 heartbeater.cc:464] Registering TS with master...
I20260929 21:00:45.359093 10151 heartbeater.cc:511] Master 127.0.97.62:35933 requested a full tablet report, sending...
I20260929 21:00:45.360473 9704 ts_manager.cc:194] Registered new tserver with Master: a48cdf649e95442ebd9cfcf8c1a1c86e (127.0.97.3:44285)
I20260929 21:00:45.361299 9704 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:50291
I20260929 21:00:45.363160 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 21:00:45.369165 9892 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:45.367709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39568 (local address 127.0.97.62:35933)
0929 21:00:45.367881 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:45.367885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:45.368370 (+ 485us) server_negotiation.cc:407] Connection header received
0929 21:00:45.368500 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:45.368504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:45.368566 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:45.368642 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:45.369011 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39568: BlockingRecv error: recv got EOF from 127.0.0.1:39568 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 21:00:45.369858 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 9763
I20260929 21:00:45.377592 10160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:45.370699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50730 (local address 127.0.97.1:44115)
0929 21:00:45.371184 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:45.371191 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:45.371215 (+ 24us) server_negotiation.cc:407] Connection header received
0929 21:00:45.371267 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:45.371272 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:45.371460 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:45.371578 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:45.371914 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:45.372932 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:45.374501 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:45.375048 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:45.375143 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:45.376134 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:45.376140 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:45.376152 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:00:45.376697 (+ 545us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:45.376740 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:00:45.376757 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:45.377025 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:45.377117 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 21:00:45.377257 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":248,"threads_started":1}
I20260929 21:00:45.378917 9885 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:00:45.461078 9890 heartbeater.cc:503] Master 127.0.97.62:35933 was elected leader, sending a full tablet report...
I20260929 21:00:45.461465 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 9763
W20260929 21:00:45.477814 9688 connection.cc:535] server connection from 127.0.97.1:35303 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 21:00:45.478075 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 9894
I20260929 21:00:45.486045 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:45.478900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46336 (local address 127.0.97.2:41065)
0929 21:00:45.479443 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:45.479450 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:45.479475 (+ 25us) server_negotiation.cc:407] Connection header received
0929 21:00:45.479534 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:45.479540 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:45.479754 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:45.479927 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:45.480355 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:45.481577 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:45.482806 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:45.483357 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:45.483493 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:45.484556 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:45.484564 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:45.484576 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:00:45.485229 (+ 653us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:45.485274 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:00:45.485293 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:45.485442 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:45.485578 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 21:00:45.485721 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":283,"threads_started":1}
I20260929 21:00:45.487251 10016 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:00:45.569797 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 9894
I20260929 21:00:45.586722 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 10024
I20260929 21:00:45.594915 10164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:45.587727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38332 (local address 127.0.97.3:44285)
0929 21:00:45.588332 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:45.588340 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:45.588369 (+ 29us) server_negotiation.cc:407] Connection header received
0929 21:00:45.588426 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:45.588432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:45.588648 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:45.588819 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:45.589287 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:45.590307 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:45.591643 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:45.592238 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:45.592378 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:45.593504 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:45.593512 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:45.593525 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:00:45.594105 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:45.594151 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:00:45.594169 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:45.594294 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:45.594440 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 21:00:45.594578 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":313,"threads_started":1}
I20260929 21:00:45.596232 10146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:00:45.678647 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 10024
I20260929 21:00:45.694810 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 9673
I20260929 21:00:45.695974 9734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:00:45.796154 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 9673
2026-09-29T21:00:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2372 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T21:00:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:00:45Z Disabled control of system clock
I20260929 21:00:45.865816 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41609
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:33347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 21:00:46.185562 10170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:46.185871 10170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:46.185963 10170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:46.193156 10170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:00:46.193259 10170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:46.193295 10170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:46.193351 10170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:00:46.193428 10170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:00:46.202489 10170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41609
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:00:46.203819 10170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:46.205534 10170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:46.217083 10175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:46.217067 10176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:46.217128 10178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:46.217782 10170 server_base.cc:1034] running on GCE node
I20260929 21:00:46.218550 10170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:46.219817 10170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:46.221019 10170 hybrid_clock.cc:648] HybridClock initialized: now 1790715646220984 us; error 27 us; skew 500 ppm
I20260929 21:00:46.221364 10170 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 21:00:46.223637 10170 webserver.cc:460] Webserver started at http://127.0.97.62:46745/ using document root <none> and password file <none>
I20260929 21:00:46.224367 10170 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:46.224479 10170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:46.224797 10170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:00:46.227622 10170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "a053d91dcc764a30a010fa5719fd6134"
format_stamp: "Formatted at 2026-09-29 21:00:46 on dist-test-slave-1dms"
server_key: "c9a574f7c09cc645ff59789ea923ab75"
server_key_iv: "58e5dace1c08a119afa724d18215232b"
server_key_version: "encryptionkey@0"
I20260929 21:00:46.228407 10170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "a053d91dcc764a30a010fa5719fd6134"
format_stamp: "Formatted at 2026-09-29 21:00:46 on dist-test-slave-1dms"
server_key: "c9a574f7c09cc645ff59789ea923ab75"
server_key_iv: "58e5dace1c08a119afa724d18215232b"
server_key_version: "encryptionkey@0"
I20260929 21:00:46.233443 10170 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260929 21:00:46.236791 10184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:46.238232 10170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 21:00:46.238413 10170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "a053d91dcc764a30a010fa5719fd6134"
format_stamp: "Formatted at 2026-09-29 21:00:46 on dist-test-slave-1dms"
server_key: "c9a574f7c09cc645ff59789ea923ab75"
server_key_iv: "58e5dace1c08a119afa724d18215232b"
server_key_version: "encryptionkey@0"
I20260929 21:00:46.238632 10170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:46.244091 10170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:46.244943 10170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:00:46.245227 10170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:46.262274 10170 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41609
I20260929 21:00:46.262306 10235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41609 every 8 connection(s)
I20260929 21:00:46.263526 10170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:00:46.265069 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 10170
I20260929 21:00:46.265290 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:00:46.266126 388 external_mini_cluster.cc:1428] Setting key e38f5eddeab6ec6fd57352b48309815f
I20260929 21:00:46.268517 10236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:00:46.275326 10239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:46.268179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45398 (local address 127.0.97.62:41609)
0929 21:00:46.269242 (+ 1063us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:46.269256 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:46.269303 (+ 47us) server_negotiation.cc:407] Connection header received
0929 21:00:46.271338 (+ 2035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:46.271365 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:46.271909 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:46.272335 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:46.273358 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:46.273367 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:46.273382 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:00:46.273397 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:46.273494 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:00:46.273514 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:46.273725 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:46.273860 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 21:00:46.274111 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":349,"threads_started":1}
I20260929 21:00:46.281752 10236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a053d91dcc764a30a010fa5719fd6134: Bootstrap starting.
I20260929 21:00:46.284788 10236 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a053d91dcc764a30a010fa5719fd6134: Neither blocks nor log segments found. Creating new log.
I20260929 21:00:46.285792 10236 log.cc:824] T 00000000000000000000000000000000 P a053d91dcc764a30a010fa5719fd6134: Log is configured to *not* fsync() on all Append() calls
I20260929 21:00:46.288355 10236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a053d91dcc764a30a010fa5719fd6134: No bootstrap required, opened a new log
I20260929 21:00:46.295332 10236 raft_consensus.cc:348] T 00000000000000000000000000000000 P a053d91dcc764a30a010fa5719fd6134 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a053d91dcc764a30a010fa5719fd6134" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41609 } }
I20260929 21:00:46.295648 10236 raft_consensus.cc:374] T 00000000000000000000000000000000 P a053d91dcc764a30a010fa5719fd6134 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:00:46.295727 10236 raft_consensus.cc:729] T 00000000000000000000000000000000 P a053d91dcc764a30a010fa5719fd6134 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a053d91dcc764a30a010fa5719fd6134, State: Initialized, Role: FOLLOWER
I20260929 21:00:46.296213 10236 consensus_queue.cc:260] T 00000000000000000000000000000000 P a053d91dcc764a30a010fa5719fd6134 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a053d91dcc764a30a010fa5719fd6134" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41609 } }
I20260929 21:00:46.296408 10236 raft_consensus.cc:388] T 00000000000000000000000000000000 P a053d91dcc764a30a010fa5719fd6134 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:00:46.296499 10236 raft_consensus.cc:482] T 00000000000000000000000000000000 P a053d91dcc764a30a010fa5719fd6134 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:00:46.296619 10236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a053d91dcc764a30a010fa5719fd6134 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:00:46.298944 10236 raft_consensus.cc:504] T 00000000000000000000000000000000 P a053d91dcc764a30a010fa5719fd6134 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a053d91dcc764a30a010fa5719fd6134" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41609 } }
I20260929 21:00:46.299379 10236 leader_election.cc:302] T 00000000000000000000000000000000 P a053d91dcc764a30a010fa5719fd6134 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a053d91dcc764a30a010fa5719fd6134; no voters:
I20260929 21:00:46.299918 10236 leader_election.cc:288] T 00000000000000000000000000000000 P a053d91dcc764a30a010fa5719fd6134 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 21:00:46.300073 10241 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a053d91dcc764a30a010fa5719fd6134 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:00:46.300782 10241 raft_consensus.cc:686] T 00000000000000000000000000000000 P a053d91dcc764a30a010fa5719fd6134 [term 1 LEADER]: Becoming Leader. State: Replica: a053d91dcc764a30a010fa5719fd6134, State: Running, Role: LEADER
I20260929 21:00:46.301379 10241 consensus_queue.cc:237] T 00000000000000000000000000000000 P a053d91dcc764a30a010fa5719fd6134 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a053d91dcc764a30a010fa5719fd6134" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41609 } }
I20260929 21:00:46.301997 10236 sys_catalog.cc:564] T 00000000000000000000000000000000 P a053d91dcc764a30a010fa5719fd6134 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:00:46.304126 10242 sys_catalog.cc:455] T 00000000000000000000000000000000 P a053d91dcc764a30a010fa5719fd6134 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a053d91dcc764a30a010fa5719fd6134" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a053d91dcc764a30a010fa5719fd6134" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41609 } } }
I20260929 21:00:46.304545 10242 sys_catalog.cc:458] T 00000000000000000000000000000000 P a053d91dcc764a30a010fa5719fd6134 [sys.catalog]: This master's current role is: LEADER
I20260929 21:00:46.306664 10250 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:00:46.306617 10243 sys_catalog.cc:455] T 00000000000000000000000000000000 P a053d91dcc764a30a010fa5719fd6134 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a053d91dcc764a30a010fa5719fd6134. Latest consensus state: current_term: 1 leader_uuid: "a053d91dcc764a30a010fa5719fd6134" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a053d91dcc764a30a010fa5719fd6134" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41609 } } }
I20260929 21:00:46.307049 10243 sys_catalog.cc:458] T 00000000000000000000000000000000 P a053d91dcc764a30a010fa5719fd6134 [sys.catalog]: This master's current role is: LEADER
I20260929 21:00:46.311064 10250 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:00:46.319192 10250 catalog_manager.cc:1347] Generated new cluster ID: 3be8bbc398cf447e8ef4e222053e3f69
I20260929 21:00:46.319311 10250 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:00:46.339238 10250 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:00:46.339694 10250 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:00:46.353840 10250 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a053d91dcc764a30a010fa5719fd6134: Generated new TSK 0
I20260929 21:00:46.354748 10250 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 21:00:46.360481 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41609
--builtin_ntp_servers=127.0.97.20:33347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 21:00:46.687202 10260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:46.687462 10260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:46.687613 10260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:46.694636 10260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:46.694727 10260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:46.694890 10260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 21:00:46.703765 10260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:00:46.705020 10260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:46.706856 10260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:46.719856 10268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:46.719887 10266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:46.720592 10260 server_base.cc:1034] running on GCE node
W20260929 21:00:46.720670 10265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:46.721351 10260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:46.722167 10260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:46.723362 10260 hybrid_clock.cc:648] HybridClock initialized: now 1790715646723320 us; error 36 us; skew 500 ppm
I20260929 21:00:46.723692 10260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 21:00:46.726724 10260 webserver.cc:460] Webserver started at http://127.0.97.1:33979/ using document root <none> and password file <none>
I20260929 21:00:46.727413 10260 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:46.727522 10260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:46.727847 10260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:00:46.730677 10260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-0/data/instance:
uuid: "77e281e5cb2f45cda5b61a6c0d897e26"
format_stamp: "Formatted at 2026-09-29 21:00:46 on dist-test-slave-1dms"
server_key: "88254f564dba7df1caad77546fee0639"
server_key_iv: "1c8234011571aed9d4350e679d4917ce"
server_key_version: "encryptionkey@0"
I20260929 21:00:46.731415 10260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-0/wal/instance:
uuid: "77e281e5cb2f45cda5b61a6c0d897e26"
format_stamp: "Formatted at 2026-09-29 21:00:46 on dist-test-slave-1dms"
server_key: "88254f564dba7df1caad77546fee0639"
server_key_iv: "1c8234011571aed9d4350e679d4917ce"
server_key_version: "encryptionkey@0"
I20260929 21:00:46.736297 10260 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 21:00:46.739583 10274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:46.741021 10260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:00:46.741195 10260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-0/wal
uuid: "77e281e5cb2f45cda5b61a6c0d897e26"
format_stamp: "Formatted at 2026-09-29 21:00:46 on dist-test-slave-1dms"
server_key: "88254f564dba7df1caad77546fee0639"
server_key_iv: "1c8234011571aed9d4350e679d4917ce"
server_key_version: "encryptionkey@0"
I20260929 21:00:46.741406 10260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:46.747102 10260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:46.747833 10260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:00:46.748124 10260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:46.749087 10260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:00:46.750432 10260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:00:46.750540 10260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:46.750677 10260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:00:46.750782 10260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:46.782393 10260 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44653
I20260929 21:00:46.782478 10386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44653 every 8 connection(s)
I20260929 21:00:46.783661 10260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
I20260929 21:00:46.784096 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 10260
I20260929 21:00:46.784312 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-0/wal/instance
I20260929 21:00:46.785212 388 external_mini_cluster.cc:1428] Setting key a20f657c679057dbe0875d7e45c42c13
I20260929 21:00:46.788156 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41609
--builtin_ntp_servers=127.0.97.20:33347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 21:00:46.792428 10389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:46.786604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41585 (local address 127.0.97.62:41609)
0929 21:00:46.787076 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:46.787084 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:46.788474 (+ 1390us) server_negotiation.cc:407] Connection header received
0929 21:00:46.790165 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:46.790172 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:46.790233 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:46.790349 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:46.791596 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:46.791604 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:46.791611 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:00:46.791633 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:46.791687 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:00:46.791704 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:46.791826 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:46.791999 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 21:00:46.792062 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":245,"threads_started":1}
I20260929 21:00:46.793191 10390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:46.787299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41609 (local address 127.0.97.1:41585)
0929 21:00:46.788331 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:46.788380 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:46.789917 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:46.790532 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:46.790539 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:46.790624 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:46.791313 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 21:00:46.791316 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:46.791352 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 21:00:46.791432 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:46.791803 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:46.791812 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:46.791818 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 21:00:46.792019 (+ 201us) client_negotiation.cc:240] Negotiation successful
0929 21:00:46.792198 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":329,"threads_started":1}
I20260929 21:00:46.794716 10387 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41609
I20260929 21:00:46.794999 10387 heartbeater.cc:464] Registering TS with master...
I20260929 21:00:46.795531 10387 heartbeater.cc:511] Master 127.0.97.62:41609 requested a full tablet report, sending...
I20260929 21:00:46.797654 10201 ts_manager.cc:194] Registered new tserver with Master: 77e281e5cb2f45cda5b61a6c0d897e26 (127.0.97.1:44653)
W20260929 21:00:47.105201 10391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:47.105497 10391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:47.105587 10391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:47.112751 10391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:47.112970 10391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:47.113152 10391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 21:00:47.122624 10391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:00:47.123811 10391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:47.125631 10391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:47.138371 10397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:47.138456 10396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:47.138468 10399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:47.142375 10391 server_base.cc:1034] running on GCE node
I20260929 21:00:47.143076 10391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:47.144280 10391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:47.145511 10391 hybrid_clock.cc:648] HybridClock initialized: now 1790715647145449 us; error 60 us; skew 500 ppm
I20260929 21:00:47.145826 10391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 21:00:47.148030 10391 webserver.cc:460] Webserver started at http://127.0.97.2:36271/ using document root <none> and password file <none>
I20260929 21:00:47.148680 10391 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:47.148763 10391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:47.149078 10391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:00:47.151826 10391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-1/data/instance:
uuid: "e97995148a9042c1a8d7823717247cb0"
format_stamp: "Formatted at 2026-09-29 21:00:47 on dist-test-slave-1dms"
server_key: "18c2d02da3c4df4697d04dc0c1cba34d"
server_key_iv: "c1fea3ba0fbea3bd10e12778bd903a1f"
server_key_version: "encryptionkey@0"
I20260929 21:00:47.152530 10391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e97995148a9042c1a8d7823717247cb0"
format_stamp: "Formatted at 2026-09-29 21:00:47 on dist-test-slave-1dms"
server_key: "18c2d02da3c4df4697d04dc0c1cba34d"
server_key_iv: "c1fea3ba0fbea3bd10e12778bd903a1f"
server_key_version: "encryptionkey@0"
I20260929 21:00:47.157483 10391 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 21:00:47.160728 10405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:47.162075 10391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 21:00:47.162252 10391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-1/wal
uuid: "e97995148a9042c1a8d7823717247cb0"
format_stamp: "Formatted at 2026-09-29 21:00:47 on dist-test-slave-1dms"
server_key: "18c2d02da3c4df4697d04dc0c1cba34d"
server_key_iv: "c1fea3ba0fbea3bd10e12778bd903a1f"
server_key_version: "encryptionkey@0"
I20260929 21:00:47.162493 10391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:47.168129 10391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:47.169003 10391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:00:47.169265 10391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:47.170313 10391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:00:47.171703 10391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:00:47.171828 10391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:47.171950 10391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:00:47.172034 10391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:47.203135 10391 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38457
I20260929 21:00:47.203168 10517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38457 every 8 connection(s)
I20260929 21:00:47.204470 10391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
I20260929 21:00:47.210886 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 10391
I20260929 21:00:47.211081 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-1/wal/instance
I20260929 21:00:47.211659 388 external_mini_cluster.cc:1428] Setting key 32e8fa0789eef56cbdfa67eaebe18967
I20260929 21:00:47.212716 10389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:47.207060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55101 (local address 127.0.97.62:41609)
0929 21:00:47.207265 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:47.207271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:47.208742 (+ 1471us) server_negotiation.cc:407] Connection header received
0929 21:00:47.210406 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:47.210411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:47.210464 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:47.210582 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:47.211969 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:47.211976 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:47.211981 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:00:47.212000 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:47.212036 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:00:47.212051 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:47.212169 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:47.212353 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 21:00:47.212420 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 21:00:47.213766 10520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:47.207645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41609 (local address 127.0.97.2:55101)
0929 21:00:47.208587 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:47.208629 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:47.210158 (+ 1529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:47.210751 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:47.210758 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:47.210867 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:47.211625 (+ 758us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 21:00:47.211627 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:47.211664 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 21:00:47.211745 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:47.212155 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:47.212162 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:47.212167 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 21:00:47.212427 (+ 260us) client_negotiation.cc:240] Negotiation successful
0929 21:00:47.212666 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":307,"threads_started":1}
I20260929 21:00:47.214524 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41609
--builtin_ntp_servers=127.0.97.20:33347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 21:00:47.215166 10518 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41609
I20260929 21:00:47.215447 10518 heartbeater.cc:464] Registering TS with master...
I20260929 21:00:47.216003 10518 heartbeater.cc:511] Master 127.0.97.62:41609 requested a full tablet report, sending...
I20260929 21:00:47.217331 10201 ts_manager.cc:194] Registered new tserver with Master: e97995148a9042c1a8d7823717247cb0 (127.0.97.2:38457)
W20260929 21:00:47.535318 10521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:47.535617 10521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:47.535779 10521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:47.542821 10521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:47.542944 10521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:47.543125 10521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 21:00:47.552136 10521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:00:47.553421 10521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:47.555114 10521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:47.568735 10526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:47.568763 10527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:47.568877 10529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:47.569528 10521 server_base.cc:1034] running on GCE node
I20260929 21:00:47.570156 10521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:47.571138 10521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:47.572368 10521 hybrid_clock.cc:648] HybridClock initialized: now 1790715647572317 us; error 48 us; skew 500 ppm
I20260929 21:00:47.572680 10521 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 21:00:47.574867 10521 webserver.cc:460] Webserver started at http://127.0.97.3:41333/ using document root <none> and password file <none>
I20260929 21:00:47.575548 10521 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:47.575675 10521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:47.575987 10521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:00:47.578788 10521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-2/data/instance:
uuid: "178c94a5679845efad40914649319424"
format_stamp: "Formatted at 2026-09-29 21:00:47 on dist-test-slave-1dms"
server_key: "d9ae847a2d0a3c91828924d6806de4e8"
server_key_iv: "999875387152d3edbfb0e0989daa49a0"
server_key_version: "encryptionkey@0"
I20260929 21:00:47.579566 10521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-2/wal/instance:
uuid: "178c94a5679845efad40914649319424"
format_stamp: "Formatted at 2026-09-29 21:00:47 on dist-test-slave-1dms"
server_key: "d9ae847a2d0a3c91828924d6806de4e8"
server_key_iv: "999875387152d3edbfb0e0989daa49a0"
server_key_version: "encryptionkey@0"
I20260929 21:00:47.584630 10521 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 21:00:47.587975 10535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:47.589291 10521 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:00:47.589468 10521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-2/wal
uuid: "178c94a5679845efad40914649319424"
format_stamp: "Formatted at 2026-09-29 21:00:47 on dist-test-slave-1dms"
server_key: "d9ae847a2d0a3c91828924d6806de4e8"
server_key_iv: "999875387152d3edbfb0e0989daa49a0"
server_key_version: "encryptionkey@0"
I20260929 21:00:47.589704 10521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:47.595117 10521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:47.595851 10521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:00:47.596107 10521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:47.597011 10521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:00:47.598320 10521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:00:47.598448 10521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:47.598579 10521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:00:47.598664 10521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:47.631098 10521 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39071
I20260929 21:00:47.631150 10647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39071 every 8 connection(s)
I20260929 21:00:47.632468 10521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
I20260929 21:00:47.635262 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 10521
I20260929 21:00:47.635479 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790715556572573-388-0/minicluster-data/ts-2/wal/instance
I20260929 21:00:47.636070 388 external_mini_cluster.cc:1428] Setting key f384ae50072016bba8a30efcaa47cec2
I20260929 21:00:47.641353 10389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:47.635083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44103 (local address 127.0.97.62:41609)
0929 21:00:47.635249 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:47.635253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:47.637687 (+ 2434us) server_negotiation.cc:407] Connection header received
0929 21:00:47.639290 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:47.639296 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:47.639359 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:47.639437 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:47.640699 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:47.640702 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:47.640705 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:00:47.640717 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:47.640741 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:00:47.640751 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:47.640885 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:47.641054 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 21:00:47.641098 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 21:00:47.642319 10650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:47.635870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41609 (local address 127.0.97.3:44103)
0929 21:00:47.637529 (+ 1659us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:47.637573 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:47.639052 (+ 1479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:47.639647 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:47.639655 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:47.639746 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:47.640439 (+ 693us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 21:00:47.640448 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:47.640485 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 21:00:47.640567 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:47.640832 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:47.640885 (+ 53us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:47.640895 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 21:00:47.641110 (+ 215us) client_negotiation.cc:240] Negotiation successful
0929 21:00:47.641297 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1095,"thread_start_us":879,"threads_started":1}
I20260929 21:00:47.643492 10648 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41609
I20260929 21:00:47.643774 10648 heartbeater.cc:464] Registering TS with master...
I20260929 21:00:47.644304 10648 heartbeater.cc:511] Master 127.0.97.62:41609 requested a full tablet report, sending...
I20260929 21:00:47.645676 10201 ts_manager.cc:194] Registered new tserver with Master: 178c94a5679845efad40914649319424 (127.0.97.3:39071)
I20260929 21:00:47.651222 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 21:00:47.657109 10389 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:00:47.655489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45414 (local address 127.0.97.62:41609)
0929 21:00:47.655637 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:47.655641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:47.656152 (+ 511us) server_negotiation.cc:407] Connection header received
0929 21:00:47.656356 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:47.656362 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:47.656425 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:47.656525 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:47.656841 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45414: BlockingRecv error: recv got EOF from 127.0.0.1:45414 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 21:00:47.657615 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 10260
I20260929 21:00:47.661062 10657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:47.658582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47694 (local address 127.0.97.1:44653)
0929 21:00:47.659118 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:47.659125 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:47.659151 (+ 26us) server_negotiation.cc:407] Connection header received
0929 21:00:47.659204 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:47.659210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:47.659420 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:47.659540 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:47.660391 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:47.660397 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:47.660409 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:00:47.660421 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:47.660458 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:00:47.660471 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:47.660570 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:47.660650 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 21:00:47.660750 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":272,"threads_started":1}
I20260929 21:00:47.662215 10382 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:00:47.742868 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 10260
I20260929 21:00:47.758764 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 10391
I20260929 21:00:47.762434 10659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:47.759711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56812 (local address 127.0.97.2:38457)
0929 21:00:47.760257 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:47.760264 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:47.760291 (+ 27us) server_negotiation.cc:407] Connection header received
0929 21:00:47.760353 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:47.760359 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:47.760576 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:47.760700 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:47.761693 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:47.761699 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:47.761712 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:00:47.761729 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:47.761770 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:00:47.761784 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:47.761909 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:47.762025 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 21:00:47.762129 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":249,"threads_started":1}
I20260929 21:00:47.763633 10513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:00:47.843987 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 10391
I20260929 21:00:47.859503 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 10521
I20260929 21:00:47.863461 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:47.860490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51634 (local address 127.0.97.3:39071)
0929 21:00:47.861200 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:47.861206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:47.861236 (+ 30us) server_negotiation.cc:407] Connection header received
0929 21:00:47.861294 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:47.861300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:47.861513 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:47.861640 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:47.862678 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:47.862684 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:47.862695 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:00:47.862706 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:47.862744 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:00:47.862756 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:47.862886 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:47.863026 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 21:00:47.863140 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":394,"threads_started":1}
I20260929 21:00:47.864701 10643 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:00:47.945791 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 10521
I20260929 21:00:47.961416 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 10170
I20260929 21:00:47.962661 10231 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:00:48.061546 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 10170
2026-09-29T21:00:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2264 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 21:00:48 dist-test-slave-1dms krb5kdc[10666](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 21:00:48 dist-test-slave-1dms krb5kdc[10666](info): set up 2 sockets
Sep 29 21:00:48 dist-test-slave-1dms krb5kdc[10666](info): commencing operation
krb5kdc: starting...
W20260929 21:00:49.157826 388 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 21:00:49 dist-test-slave-1dms krb5kdc[10666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T21:00:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:00:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 21:00:49.381225 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38829
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:00:49.695873 10679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:49.696208 10679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:49.696364 10679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:49.703490 10679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:00:49.703632 10679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:49.703732 10679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:49.703837 10679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:00:49.703912 10679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:00:49.713645 10679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:00:49.714931 10679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:49.716282 10679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:49.727308 10684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:49.727408 10685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:49.727382 10687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:49.728439 10679 server_base.cc:1034] running on GCE node
I20260929 21:00:49.729245 10679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:49.730659 10679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:49.731890 10679 hybrid_clock.cc:648] HybridClock initialized: now 1790715649731853 us; error 44 us; skew 500 ppm
I20260929 21:00:49.732213 10679 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:00:49 dist-test-slave-1dms krb5kdc[10666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:00:49.744174 10679 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 21:00:49.745625 10679 webserver.cc:460] Webserver started at http://127.0.97.62:36609/ using document root <none> and password file <none>
I20260929 21:00:49.746294 10679 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:49.746446 10679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:49.746737 10679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:00:49.749603 10679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "69967fdb7413412281dfd30378ada995"
format_stamp: "Formatted at 2026-09-29 21:00:49 on dist-test-slave-1dms"
server_key: "7b2b76202fbb6ec3abad34205887250b"
server_key_iv: "150d33b59e414e762e7eadb0bde5b625"
server_key_version: "encryptionkey@0"
I20260929 21:00:49.750349 10679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "69967fdb7413412281dfd30378ada995"
format_stamp: "Formatted at 2026-09-29 21:00:49 on dist-test-slave-1dms"
server_key: "7b2b76202fbb6ec3abad34205887250b"
server_key_iv: "150d33b59e414e762e7eadb0bde5b625"
server_key_version: "encryptionkey@0"
I20260929 21:00:49.755079 10679 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 21:00:49.758272 10694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:49.759510 10679 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 21:00:49.759683 10679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "69967fdb7413412281dfd30378ada995"
format_stamp: "Formatted at 2026-09-29 21:00:49 on dist-test-slave-1dms"
server_key: "7b2b76202fbb6ec3abad34205887250b"
server_key_iv: "150d33b59e414e762e7eadb0bde5b625"
server_key_version: "encryptionkey@0"
I20260929 21:00:49.759859 10679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:49.787063 10679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:49.790812 10679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:00:49.791114 10679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:49.808779 10679 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38829
I20260929 21:00:49.808811 10745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38829 every 8 connection(s)
I20260929 21:00:49.810132 10679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:00:49.811326 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 10679
I20260929 21:00:49.811563 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:00:49.812319 388 external_mini_cluster.cc:1428] Setting key 51015c0a059144e981871e0a72ad0f21
I20260929 21:00:49.814715 10746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 21:00:49 dist-test-slave-1dms krb5kdc[10666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:00:49.827411 10746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69967fdb7413412281dfd30378ada995: Bootstrap starting.
I20260929 21:00:49.830646 10746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69967fdb7413412281dfd30378ada995: Neither blocks nor log segments found. Creating new log.
I20260929 21:00:49.831601 10746 log.cc:824] T 00000000000000000000000000000000 P 69967fdb7413412281dfd30378ada995: Log is configured to *not* fsync() on all Append() calls
I20260929 21:00:49.834196 10749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:49.814357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46804 (local address 127.0.97.62:38829)
0929 21:00:49.815114 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:49.815125 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:49.815163 (+ 38us) server_negotiation.cc:407] Connection header received
0929 21:00:49.816687 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:49.816705 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:49.817145 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:49.817467 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:49.819250 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:49.820552 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:49.822288 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:49.822914 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:49.823038 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:49.827557 (+ 4519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:49.827567 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:49.827585 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:49.827626 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:49.830546 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:49.831210 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:49.831225 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:49.831231 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:49.831388 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:49.831755 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:49.831761 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:49.831764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:49.832231 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:49.832535 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:49.832956 (+ 421us) server_negotiation.cc:299] Negotiation successful
0929 21:00:49.833323 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":347,"threads_started":1}
I20260929 21:00:49.834473 10746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69967fdb7413412281dfd30378ada995: No bootstrap required, opened a new log
I20260929 21:00:49.841684 10746 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69967fdb7413412281dfd30378ada995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69967fdb7413412281dfd30378ada995" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38829 } }
I20260929 21:00:49.842077 10746 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69967fdb7413412281dfd30378ada995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:00:49.842171 10746 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69967fdb7413412281dfd30378ada995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69967fdb7413412281dfd30378ada995, State: Initialized, Role: FOLLOWER
I20260929 21:00:49.842684 10746 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69967fdb7413412281dfd30378ada995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69967fdb7413412281dfd30378ada995" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38829 } }
I20260929 21:00:49.842882 10746 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69967fdb7413412281dfd30378ada995 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:00:49.842980 10746 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69967fdb7413412281dfd30378ada995 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:00:49.843101 10746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69967fdb7413412281dfd30378ada995 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:00:49.845400 10746 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69967fdb7413412281dfd30378ada995 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69967fdb7413412281dfd30378ada995" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38829 } }
I20260929 21:00:49.845822 10746 leader_election.cc:302] T 00000000000000000000000000000000 P 69967fdb7413412281dfd30378ada995 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69967fdb7413412281dfd30378ada995; no voters:
I20260929 21:00:49.846306 10746 leader_election.cc:288] T 00000000000000000000000000000000 P 69967fdb7413412281dfd30378ada995 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 21:00:49.846437 10751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69967fdb7413412281dfd30378ada995 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:00:49.847143 10751 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69967fdb7413412281dfd30378ada995 [term 1 LEADER]: Becoming Leader. State: Replica: 69967fdb7413412281dfd30378ada995, State: Running, Role: LEADER
I20260929 21:00:49.847673 10751 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69967fdb7413412281dfd30378ada995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69967fdb7413412281dfd30378ada995" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38829 } }
I20260929 21:00:49.848351 10746 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69967fdb7413412281dfd30378ada995 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:00:49.850457 10753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69967fdb7413412281dfd30378ada995 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69967fdb7413412281dfd30378ada995. Latest consensus state: current_term: 1 leader_uuid: "69967fdb7413412281dfd30378ada995" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69967fdb7413412281dfd30378ada995" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38829 } } }
I20260929 21:00:49.850546 10752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69967fdb7413412281dfd30378ada995 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69967fdb7413412281dfd30378ada995" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69967fdb7413412281dfd30378ada995" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38829 } } }
I20260929 21:00:49.850878 10752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69967fdb7413412281dfd30378ada995 [sys.catalog]: This master's current role is: LEADER
I20260929 21:00:49.850883 10753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69967fdb7413412281dfd30378ada995 [sys.catalog]: This master's current role is: LEADER
I20260929 21:00:49.852550 10760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:00:49.857537 10760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:00:49.865952 10760 catalog_manager.cc:1347] Generated new cluster ID: d1ee4d7af0c04adc841c4996c784684a
I20260929 21:00:49.866108 10760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:00:49.880918 10760 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:00:49.882201 10760 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:00:49.895596 10760 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69967fdb7413412281dfd30378ada995: Generated new TSK 0
I20260929 21:00:49.896430 10760 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 21:00:49.964680 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38829
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:00:50.290197 10772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:50.290565 10772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:50.290778 10772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:50.298348 10772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:50.298444 10772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:50.298554 10772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 21:00:50.308025 10772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:00:50.309374 10772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:50.310635 10772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:50.323570 10778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:50.323541 10777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:50.324241 10780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:50.324635 10772 server_base.cc:1034] running on GCE node
I20260929 21:00:50.325309 10772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:50.326264 10772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:50.327481 10772 hybrid_clock.cc:648] HybridClock initialized: now 1790715650327437 us; error 60 us; skew 500 ppm
I20260929 21:00:50.327793 10772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:00:50 dist-test-slave-1dms krb5kdc[10666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:00:50.330858 10772 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 21:00:50.332209 10772 webserver.cc:460] Webserver started at http://127.0.97.1:33151/ using document root <none> and password file <none>
I20260929 21:00:50.332973 10772 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:50.333114 10772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:50.333508 10772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:00:50.336651 10772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-0/data/instance:
uuid: "d6929b0fa8784b10a4c6584a7be75879"
format_stamp: "Formatted at 2026-09-29 21:00:50 on dist-test-slave-1dms"
server_key: "a156ff6d34d5aba4467250cba6150c00"
server_key_iv: "5e843b22568c5e138b44e52c78e39559"
server_key_version: "encryptionkey@0"
I20260929 21:00:50.337404 10772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d6929b0fa8784b10a4c6584a7be75879"
format_stamp: "Formatted at 2026-09-29 21:00:50 on dist-test-slave-1dms"
server_key: "a156ff6d34d5aba4467250cba6150c00"
server_key_iv: "5e843b22568c5e138b44e52c78e39559"
server_key_version: "encryptionkey@0"
I20260929 21:00:50.342243 10772 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 21:00:50.345417 10787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:50.346705 10772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 21:00:50.346897 10772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-0/wal
uuid: "d6929b0fa8784b10a4c6584a7be75879"
format_stamp: "Formatted at 2026-09-29 21:00:50 on dist-test-slave-1dms"
server_key: "a156ff6d34d5aba4467250cba6150c00"
server_key_iv: "5e843b22568c5e138b44e52c78e39559"
server_key_version: "encryptionkey@0"
I20260929 21:00:50.347088 10772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:50.363526 10772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:50.367224 10772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:00:50.367516 10772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:50.368381 10772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:00:50.369681 10772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:00:50.369782 10772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:50.369936 10772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:00:50.370008 10772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:50.402572 10772 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33433
I20260929 21:00:50.402621 10899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33433 every 8 connection(s)
I20260929 21:00:50.403832 10772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
I20260929 21:00:50.408975 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 10772
I20260929 21:00:50.409188 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-0/wal/instance
I20260929 21:00:50.409780 388 external_mini_cluster.cc:1428] Setting key 8b7cd5471eff818e6c587ae18c3f262a
Sep 29 21:00:50 dist-test-slave-1dms krb5kdc[10666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:00:50.424167 10902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:50.406277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46707 (local address 127.0.97.62:38829)
0929 21:00:50.406695 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:50.406699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:50.407703 (+ 1004us) server_negotiation.cc:407] Connection header received
0929 21:00:50.409678 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:50.409683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:50.409780 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:50.409905 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:50.411503 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:50.412539 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:50.414104 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:50.414776 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:50.414921 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:50.418344 (+ 3423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:50.418354 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:50.418362 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:50.418405 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:50.421106 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:50.422015 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:50.422019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:50.422022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:50.422128 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:50.422584 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:50.422587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:50.422589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:50.422815 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:50.422915 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:50.423744 (+ 829us) server_negotiation.cc:299] Negotiation successful
0929 21:00:50.423909 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":219,"threads_started":1}
I20260929 21:00:50.425690 10903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:50.406874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38829 (local address 127.0.97.1:46707)
0929 21:00:50.407552 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:50.407588 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:50.409423 (+ 1835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:50.410147 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:50.410156 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:50.410779 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:50.411307 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:50.411354 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:50.412700 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:50.412704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:50.413955 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:50.413966 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:50.415113 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:50.415119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:50.415314 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:50.415985 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:50.416010 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:50.418126 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:50.421235 (+ 3109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:50.421246 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:50.421267 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:50.421896 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:50.422229 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:50.422236 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:50.422243 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:50.422478 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:50.422941 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:50.422951 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:50.423377 (+ 426us) client_negotiation.cc:769] Sending connection context
0929 21:00:50.423664 (+ 287us) client_negotiation.cc:240] Negotiation successful
0929 21:00:50.424213 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":270,"threads_started":1}
I20260929 21:00:50.427314 10900 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38829
I20260929 21:00:50.427613 10900 heartbeater.cc:464] Registering TS with master...
I20260929 21:00:50.428323 10900 heartbeater.cc:511] Master 127.0.97.62:38829 requested a full tablet report, sending...
I20260929 21:00:50.430771 10711 ts_manager.cc:194] Registered new tserver with Master: d6929b0fa8784b10a4c6584a7be75879 (127.0.97.1:33433)
I20260929 21:00:50.432943 10711 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46707
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 21:00:50.463985 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38829
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:00:50.786262 10906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:50.786525 10906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:50.786685 10906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:50.793648 10906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:50.793751 10906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:50.793959 10906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 21:00:50.803212 10906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:00:50.804397 10906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:50.805672 10906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:50.819301 10911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:50.819322 10912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:50.819679 10914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:50.820698 10906 server_base.cc:1034] running on GCE node
I20260929 21:00:50.821411 10906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:50.822319 10906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:50.823539 10906 hybrid_clock.cc:648] HybridClock initialized: now 1790715650823483 us; error 50 us; skew 500 ppm
I20260929 21:00:50.823843 10906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:00:50 dist-test-slave-1dms krb5kdc[10666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:00:50.826942 10906 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 21:00:50.828322 10906 webserver.cc:460] Webserver started at http://127.0.97.2:43329/ using document root <none> and password file <none>
I20260929 21:00:50.829080 10906 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:50.829217 10906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:50.829509 10906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:00:50.832749 10906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-1/data/instance:
uuid: "50461a7d5f5542f58c69ce683bc81deb"
format_stamp: "Formatted at 2026-09-29 21:00:50 on dist-test-slave-1dms"
server_key: "69c80e835cb1a7dfc757a9a081fa68d1"
server_key_iv: "1c990b0830b46ccbb6c9d295d401e220"
server_key_version: "encryptionkey@0"
I20260929 21:00:50.833603 10906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-1/wal/instance:
uuid: "50461a7d5f5542f58c69ce683bc81deb"
format_stamp: "Formatted at 2026-09-29 21:00:50 on dist-test-slave-1dms"
server_key: "69c80e835cb1a7dfc757a9a081fa68d1"
server_key_iv: "1c990b0830b46ccbb6c9d295d401e220"
server_key_version: "encryptionkey@0"
I20260929 21:00:50.838356 10906 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 21:00:50.841548 10921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:50.842783 10906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 21:00:50.842947 10906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-1/wal
uuid: "50461a7d5f5542f58c69ce683bc81deb"
format_stamp: "Formatted at 2026-09-29 21:00:50 on dist-test-slave-1dms"
server_key: "69c80e835cb1a7dfc757a9a081fa68d1"
server_key_iv: "1c990b0830b46ccbb6c9d295d401e220"
server_key_version: "encryptionkey@0"
I20260929 21:00:50.843128 10906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:50.873140 10906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:50.877436 10906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:00:50.877727 10906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:50.878652 10906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:00:50.879976 10906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:00:50.880077 10906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:50.880231 10906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:00:50.880303 10906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:50.912565 10906 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41161
I20260929 21:00:50.912628 11033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41161 every 8 connection(s)
I20260929 21:00:50.913847 10906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
I20260929 21:00:50.914750 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 10906
I20260929 21:00:50.914947 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-1/wal/instance
I20260929 21:00:50.915486 388 external_mini_cluster.cc:1428] Setting key 43e224a9769b8df5ed7d838aabd042fb
Sep 29 21:00:50 dist-test-slave-1dms krb5kdc[10666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:00:50.935806 10902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:50.916678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46439 (local address 127.0.97.62:38829)
0929 21:00:50.916888 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:50.916894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:50.918346 (+ 1452us) server_negotiation.cc:407] Connection header received
0929 21:00:50.921197 (+ 2851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:50.921201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:50.921261 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:50.921346 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:50.923340 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:50.924021 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:50.926491 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:50.926911 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:50.927048 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:50.930388 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:50.930392 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:50.930396 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:50.930425 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:50.932929 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:50.933613 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:50.933619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:50.933623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:50.933748 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:50.934166 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:50.934171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:50.934174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:50.934429 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:50.934583 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:50.935223 (+ 640us) server_negotiation.cc:299] Negotiation successful
0929 21:00:50.935440 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 21:00:50.936735 11036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:50.917247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38829 (local address 127.0.97.2:46439)
0929 21:00:50.918158 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:50.918215 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:50.920914 (+ 2699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:50.921608 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:50.921622 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:50.922364 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:50.923156 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:50.923183 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:50.924198 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:50.924206 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:50.926344 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:50.926355 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:50.927137 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:50.927145 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:50.927376 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:50.928051 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:50.928072 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:50.930173 (+ 2101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:50.933066 (+ 2893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:50.933074 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:50.933089 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:50.933488 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:50.933857 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:50.933864 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:50.933868 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:50.934050 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:50.934557 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:50.934564 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:50.934877 (+ 313us) client_negotiation.cc:769] Sending connection context
0929 21:00:50.935133 (+ 256us) client_negotiation.cc:240] Negotiation successful
0929 21:00:50.935632 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":380,"threads_started":1}
I20260929 21:00:50.938174 11034 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38829
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 21:00:50.938479 11034 heartbeater.cc:464] Registering TS with master...
I20260929 21:00:50.939148 11034 heartbeater.cc:511] Master 127.0.97.62:38829 requested a full tablet report, sending...
I20260929 21:00:50.940610 10711 ts_manager.cc:194] Registered new tserver with Master: 50461a7d5f5542f58c69ce683bc81deb (127.0.97.2:41161)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 21:00:50.941514 10711 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:46439
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 21:00:50.967936 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38829
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:00:51.281512 11039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:00:51.281780 11039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:00:51.281880 11039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:00:51.288786 11039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:00:51.288908 11039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:00:51.289021 11039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 21:00:51.298379 11039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:00:51.299552 11039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:00:51.300901 11039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:00:51.313562 11044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:51.313586 11047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:00:51.313647 11045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:00:51.314431 11039 server_base.cc:1034] running on GCE node
I20260929 21:00:51.315032 11039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:00:51.315928 11039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:00:51.317109 11039 hybrid_clock.cc:648] HybridClock initialized: now 1790715651317077 us; error 32 us; skew 500 ppm
I20260929 21:00:51.317420 11039 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:00:51 dist-test-slave-1dms krb5kdc[10666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:00:51.320657 11039 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 21:00:51.322146 11039 webserver.cc:460] Webserver started at http://127.0.97.3:40961/ using document root <none> and password file <none>
I20260929 21:00:51.322966 11039 fs_manager.cc:359] Metadata directory not provided
I20260929 21:00:51.323072 11039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:00:51.323374 11039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:00:51.326422 11039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-2/data/instance:
uuid: "76c69bf8312c409ba42ff602b95cb83a"
format_stamp: "Formatted at 2026-09-29 21:00:51 on dist-test-slave-1dms"
server_key: "fb6af5edbdcc33ec4f818b2756b4742f"
server_key_iv: "62eacb89967195c2898722c439ac6558"
server_key_version: "encryptionkey@0"
I20260929 21:00:51.327131 11039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-2/wal/instance:
uuid: "76c69bf8312c409ba42ff602b95cb83a"
format_stamp: "Formatted at 2026-09-29 21:00:51 on dist-test-slave-1dms"
server_key: "fb6af5edbdcc33ec4f818b2756b4742f"
server_key_iv: "62eacb89967195c2898722c439ac6558"
server_key_version: "encryptionkey@0"
I20260929 21:00:51.332070 11039 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260929 21:00:51.335469 11054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:51.336791 11039 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 21:00:51.337038 11039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-2/wal
uuid: "76c69bf8312c409ba42ff602b95cb83a"
format_stamp: "Formatted at 2026-09-29 21:00:51 on dist-test-slave-1dms"
server_key: "fb6af5edbdcc33ec4f818b2756b4742f"
server_key_iv: "62eacb89967195c2898722c439ac6558"
server_key_version: "encryptionkey@0"
I20260929 21:00:51.337229 11039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:00:51.356048 11039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:00:51.359797 11039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:00:51.360076 11039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:00:51.361083 11039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:00:51.362345 11039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:00:51.362416 11039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:51.362500 11039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:00:51.362545 11039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:00:51.394827 11039 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33557
I20260929 21:00:51.394876 11166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33557 every 8 connection(s)
I20260929 21:00:51.396066 11039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
I20260929 21:00:51.397984 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 11039
I20260929 21:00:51.398195 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-2/wal/instance
I20260929 21:00:51.398638 388 external_mini_cluster.cc:1428] Setting key d140dfc797e619c665aba10d7c9e5e05
Sep 29 21:00:51 dist-test-slave-1dms krb5kdc[10666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:00:51.416945 10902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:51.398862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39711 (local address 127.0.97.62:38829)
0929 21:00:51.399078 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:51.399084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:51.400444 (+ 1360us) server_negotiation.cc:407] Connection header received
0929 21:00:51.402481 (+ 2037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:51.402485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:51.402545 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:51.402621 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:51.404523 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:51.405297 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:51.407488 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:51.407867 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:51.407955 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:51.411184 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:51.411188 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:51.411191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:51.411221 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:51.414091 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:51.414753 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:51.414761 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:51.414767 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:51.414895 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:51.415296 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:51.415303 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:51.415307 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:51.415564 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:51.415717 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:51.416380 (+ 663us) server_negotiation.cc:299] Negotiation successful
0929 21:00:51.416586 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 21:00:51.417863 11169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:51.399486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38829 (local address 127.0.97.3:39711)
0929 21:00:51.400286 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:51.400351 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:51.402197 (+ 1846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:51.402892 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:51.402903 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:51.403626 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:00:51.404364 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:51.404391 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:51.405442 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:51.405447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:51.407336 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:51.407350 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:51.408065 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:51.408070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:51.408266 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:51.408917 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:00:51.408944 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:00:51.411001 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:00:51.414225 (+ 3224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:51.414234 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:51.414248 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:51.414632 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:51.414999 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:00:51.415005 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:00:51.415009 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:00:51.415186 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:00:51.415689 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:00:51.415697 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:00:51.416014 (+ 317us) client_negotiation.cc:769] Sending connection context
0929 21:00:51.416278 (+ 264us) client_negotiation.cc:240] Negotiation successful
0929 21:00:51.416760 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":349,"threads_started":1}
I20260929 21:00:51.419056 11167 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38829
I20260929 21:00:51.419346 11167 heartbeater.cc:464] Registering TS with master...
I20260929 21:00:51.420054 11167 heartbeater.cc:511] Master 127.0.97.62:38829 requested a full tablet report, sending...
I20260929 21:00:51.421458 10711 ts_manager.cc:194] Registered new tserver with Master: 76c69bf8312c409ba42ff602b95cb83a (127.0.97.3:33557)
I20260929 21:00:51.422236 10711 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39711
I20260929 21:00:51.425738 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 21:00:51.435890 10900 heartbeater.cc:503] Master 127.0.97.62:38829 was elected leader, sending a full tablet report...
I20260929 21:00:51.442209 10902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:51.430175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46818 (local address 127.0.97.62:38829)
0929 21:00:51.430325 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:51.430329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:51.430788 (+ 459us) server_negotiation.cc:407] Connection header received
0929 21:00:51.430919 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:51.430922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:51.430979 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:51.431060 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:51.432356 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:51.433117 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:51.434270 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:51.434646 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:51.434735 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:51.437164 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:51.437168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:51.437172 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:51.437201 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:51.439661 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:51.440340 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:51.440344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:51.440346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:51.440468 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:51.440826 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:51.440829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:51.440832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:51.441186 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:51.441411 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:51.441821 (+ 410us) server_negotiation.cc:299] Negotiation successful
0929 21:00:51.441988 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 21:00:51.446169 10711 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 21:00:51.453914 10711 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 21:00:51.476342 11179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:51.465864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41161 (local address 127.0.0.1:48040)
0929 21:00:51.467109 (+ 1245us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:51.467163 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:51.467365 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:51.468523 (+ 1158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:51.468534 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:51.468574 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:00:51.469104 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:51.469118 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:51.471026 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:51.471030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:51.473443 (+ 2413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:51.473470 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:51.475530 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:51.475536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:51.475742 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:51.475798 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 21:00:51.475931 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 21:00:51.476098 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1006,"thread_start_us":864,"threads_started":1}
I20260929 21:00:51.477373 11180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:51.466096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48040 (local address 127.0.97.2:41161)
0929 21:00:51.466786 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:51.466796 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:51.467285 (+ 489us) server_negotiation.cc:407] Connection header received
0929 21:00:51.467619 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:51.467632 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:51.467947 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:51.468149 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:00:51.469311 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:51.470821 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:51.473621 (+ 2800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:51.475069 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:51.475186 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:51.475391 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:51.476874 (+ 1483us) server_negotiation.cc:299] Negotiation successful
0929 21:00:51.477059 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":340,"threads_started":1}
I20260929 21:00:51.477523 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:51.466450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33557 (local address 127.0.0.1:49736)
0929 21:00:51.469395 (+ 2945us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:51.469425 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:51.469652 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:51.470245 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:51.470252 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:51.470284 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:00:51.471030 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:51.471043 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:51.472830 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:51.472867 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:51.474800 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:51.474814 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:51.476640 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:51.476650 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:51.476862 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:51.476896 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 21:00:51.477020 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 21:00:51.477135 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2791,"mutex_wait_us":1,"thread_start_us":2624,"threads_started":1}
I20260929 21:00:51.477603 11182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:51.466991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49736 (local address 127.0.97.3:33557)
0929 21:00:51.467780 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:51.467791 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:51.469534 (+ 1743us) server_negotiation.cc:407] Connection header received
0929 21:00:51.469839 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:51.469847 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:51.470059 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:51.470238 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:00:51.471247 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:51.472701 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:51.474972 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:51.476345 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:51.476502 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:51.476717 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:51.477089 (+ 372us) server_negotiation.cc:299] Negotiation successful
0929 21:00:51.477261 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":456,"threads_started":1}
I20260929 21:00:51.478884 11183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:51.467448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33433 (local address 127.0.0.1:60062)
0929 21:00:51.471116 (+ 3668us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:51.471136 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:51.471378 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:51.472193 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:51.472197 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:51.472221 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:00:51.472613 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:51.472623 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:51.474210 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:51.474213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:51.476078 (+ 1865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:51.476092 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:51.478194 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:51.478198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:51.478307 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:51.478327 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 21:00:51.478438 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 21:00:51.478553 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3529,"thread_start_us":3334,"threads_started":1}
I20260929 21:00:51.481201 10969 tablet_service.cc:1467] Processing CreateTablet for tablet a5e65ee009cc4291a1dad3f1e23bfd45 (DEFAULT_TABLE table=test-table [id=60db4c98ab8348e1b4258abb1c6fc1b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:00:51.481200 11102 tablet_service.cc:1467] Processing CreateTablet for tablet a5e65ee009cc4291a1dad3f1e23bfd45 (DEFAULT_TABLE table=test-table [id=60db4c98ab8348e1b4258abb1c6fc1b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:00:51.482316 11102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5e65ee009cc4291a1dad3f1e23bfd45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:00:51.482321 10969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5e65ee009cc4291a1dad3f1e23bfd45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:00:51.484511 11184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:51.467892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60062 (local address 127.0.97.1:33433)
0929 21:00:51.468469 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:51.468480 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:51.471254 (+ 2774us) server_negotiation.cc:407] Connection header received
0929 21:00:51.471563 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:51.471572 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:51.471786 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:51.471977 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:00:51.472825 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:51.474074 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:51.476260 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:51.477774 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:51.477892 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:51.478093 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:51.478485 (+ 392us) server_negotiation.cc:299] Negotiation successful
0929 21:00:51.484198 (+ 5713us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":259,"threads_started":1}
I20260929 21:00:51.488015 10835 tablet_service.cc:1467] Processing CreateTablet for tablet a5e65ee009cc4291a1dad3f1e23bfd45 (DEFAULT_TABLE table=test-table [id=60db4c98ab8348e1b4258abb1c6fc1b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:00:51.489346 10835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5e65ee009cc4291a1dad3f1e23bfd45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:00:51.493321 11185 tablet_bootstrap.cc:492] T a5e65ee009cc4291a1dad3f1e23bfd45 P 76c69bf8312c409ba42ff602b95cb83a: Bootstrap starting.
I20260929 21:00:51.495949 11186 tablet_bootstrap.cc:492] T a5e65ee009cc4291a1dad3f1e23bfd45 P 50461a7d5f5542f58c69ce683bc81deb: Bootstrap starting.
I20260929 21:00:51.496794 11185 tablet_bootstrap.cc:654] T a5e65ee009cc4291a1dad3f1e23bfd45 P 76c69bf8312c409ba42ff602b95cb83a: Neither blocks nor log segments found. Creating new log.
I20260929 21:00:51.498111 11185 log.cc:824] T a5e65ee009cc4291a1dad3f1e23bfd45 P 76c69bf8312c409ba42ff602b95cb83a: Log is configured to *not* fsync() on all Append() calls
I20260929 21:00:51.499300 11186 tablet_bootstrap.cc:654] T a5e65ee009cc4291a1dad3f1e23bfd45 P 50461a7d5f5542f58c69ce683bc81deb: Neither blocks nor log segments found. Creating new log.
I20260929 21:00:51.500523 11186 log.cc:824] T a5e65ee009cc4291a1dad3f1e23bfd45 P 50461a7d5f5542f58c69ce683bc81deb: Log is configured to *not* fsync() on all Append() calls
I20260929 21:00:51.501577 11185 tablet_bootstrap.cc:492] T a5e65ee009cc4291a1dad3f1e23bfd45 P 76c69bf8312c409ba42ff602b95cb83a: No bootstrap required, opened a new log
I20260929 21:00:51.501914 11185 ts_tablet_manager.cc:1397] T a5e65ee009cc4291a1dad3f1e23bfd45 P 76c69bf8312c409ba42ff602b95cb83a: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 21:00:51.502449 11189 tablet_bootstrap.cc:492] T a5e65ee009cc4291a1dad3f1e23bfd45 P d6929b0fa8784b10a4c6584a7be75879: Bootstrap starting.
I20260929 21:00:51.505272 11186 tablet_bootstrap.cc:492] T a5e65ee009cc4291a1dad3f1e23bfd45 P 50461a7d5f5542f58c69ce683bc81deb: No bootstrap required, opened a new log
I20260929 21:00:51.505285 11189 tablet_bootstrap.cc:654] T a5e65ee009cc4291a1dad3f1e23bfd45 P d6929b0fa8784b10a4c6584a7be75879: Neither blocks nor log segments found. Creating new log.
I20260929 21:00:51.505538 11186 ts_tablet_manager.cc:1397] T a5e65ee009cc4291a1dad3f1e23bfd45 P 50461a7d5f5542f58c69ce683bc81deb: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260929 21:00:51.506500 11189 log.cc:824] T a5e65ee009cc4291a1dad3f1e23bfd45 P d6929b0fa8784b10a4c6584a7be75879: Log is configured to *not* fsync() on all Append() calls
I20260929 21:00:51.509122 11189 tablet_bootstrap.cc:492] T a5e65ee009cc4291a1dad3f1e23bfd45 P d6929b0fa8784b10a4c6584a7be75879: No bootstrap required, opened a new log
I20260929 21:00:51.509364 11189 ts_tablet_manager.cc:1397] T a5e65ee009cc4291a1dad3f1e23bfd45 P d6929b0fa8784b10a4c6584a7be75879: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20260929 21:00:51.513176 11185 raft_consensus.cc:348] T a5e65ee009cc4291a1dad3f1e23bfd45 P 76c69bf8312c409ba42ff602b95cb83a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50461a7d5f5542f58c69ce683bc81deb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41161 } } peers { permanent_uuid: "d6929b0fa8784b10a4c6584a7be75879" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33433 } } peers { permanent_uuid: "76c69bf8312c409ba42ff602b95cb83a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33557 } }
I20260929 21:00:51.513595 11185 raft_consensus.cc:374] T a5e65ee009cc4291a1dad3f1e23bfd45 P 76c69bf8312c409ba42ff602b95cb83a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:00:51.513696 11185 raft_consensus.cc:729] T a5e65ee009cc4291a1dad3f1e23bfd45 P 76c69bf8312c409ba42ff602b95cb83a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c69bf8312c409ba42ff602b95cb83a, State: Initialized, Role: FOLLOWER
I20260929 21:00:51.514317 11185 consensus_queue.cc:260] T a5e65ee009cc4291a1dad3f1e23bfd45 P 76c69bf8312c409ba42ff602b95cb83a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50461a7d5f5542f58c69ce683bc81deb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41161 } } peers { permanent_uuid: "d6929b0fa8784b10a4c6584a7be75879" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33433 } } peers { permanent_uuid: "76c69bf8312c409ba42ff602b95cb83a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33557 } }
I20260929 21:00:51.515422 11167 heartbeater.cc:503] Master 127.0.97.62:38829 was elected leader, sending a full tablet report...
I20260929 21:00:51.516081 11185 ts_tablet_manager.cc:1428] T a5e65ee009cc4291a1dad3f1e23bfd45 P 76c69bf8312c409ba42ff602b95cb83a: Time spent starting tablet: real 0.014s user 0.009s sys 0.003s
I20260929 21:00:51.515899 11186 raft_consensus.cc:348] T a5e65ee009cc4291a1dad3f1e23bfd45 P 50461a7d5f5542f58c69ce683bc81deb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50461a7d5f5542f58c69ce683bc81deb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41161 } } peers { permanent_uuid: "d6929b0fa8784b10a4c6584a7be75879" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33433 } } peers { permanent_uuid: "76c69bf8312c409ba42ff602b95cb83a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33557 } }
I20260929 21:00:51.516299 11186 raft_consensus.cc:374] T a5e65ee009cc4291a1dad3f1e23bfd45 P 50461a7d5f5542f58c69ce683bc81deb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:00:51.516389 11186 raft_consensus.cc:729] T a5e65ee009cc4291a1dad3f1e23bfd45 P 50461a7d5f5542f58c69ce683bc81deb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50461a7d5f5542f58c69ce683bc81deb, State: Initialized, Role: FOLLOWER
I20260929 21:00:51.517045 11186 consensus_queue.cc:260] T a5e65ee009cc4291a1dad3f1e23bfd45 P 50461a7d5f5542f58c69ce683bc81deb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50461a7d5f5542f58c69ce683bc81deb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41161 } } peers { permanent_uuid: "d6929b0fa8784b10a4c6584a7be75879" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33433 } } peers { permanent_uuid: "76c69bf8312c409ba42ff602b95cb83a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33557 } }
I20260929 21:00:51.517198 11189 raft_consensus.cc:348] T a5e65ee009cc4291a1dad3f1e23bfd45 P d6929b0fa8784b10a4c6584a7be75879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50461a7d5f5542f58c69ce683bc81deb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41161 } } peers { permanent_uuid: "d6929b0fa8784b10a4c6584a7be75879" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33433 } } peers { permanent_uuid: "76c69bf8312c409ba42ff602b95cb83a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33557 } }
I20260929 21:00:51.517578 11189 raft_consensus.cc:374] T a5e65ee009cc4291a1dad3f1e23bfd45 P d6929b0fa8784b10a4c6584a7be75879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:00:51.517680 11189 raft_consensus.cc:729] T a5e65ee009cc4291a1dad3f1e23bfd45 P d6929b0fa8784b10a4c6584a7be75879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6929b0fa8784b10a4c6584a7be75879, State: Initialized, Role: FOLLOWER
I20260929 21:00:51.518486 11034 heartbeater.cc:503] Master 127.0.97.62:38829 was elected leader, sending a full tablet report...
I20260929 21:00:51.518288 11189 consensus_queue.cc:260] T a5e65ee009cc4291a1dad3f1e23bfd45 P d6929b0fa8784b10a4c6584a7be75879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50461a7d5f5542f58c69ce683bc81deb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41161 } } peers { permanent_uuid: "d6929b0fa8784b10a4c6584a7be75879" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33433 } } peers { permanent_uuid: "76c69bf8312c409ba42ff602b95cb83a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33557 } }
I20260929 21:00:51.519963 11189 ts_tablet_manager.cc:1428] T a5e65ee009cc4291a1dad3f1e23bfd45 P d6929b0fa8784b10a4c6584a7be75879: Time spent starting tablet: real 0.010s user 0.007s sys 0.003s
I20260929 21:00:51.520360 11186 ts_tablet_manager.cc:1428] T a5e65ee009cc4291a1dad3f1e23bfd45 P 50461a7d5f5542f58c69ce683bc81deb: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
I20260929 21:00:51.605973 11191 raft_consensus.cc:482] T a5e65ee009cc4291a1dad3f1e23bfd45 P 76c69bf8312c409ba42ff602b95cb83a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 21:00:51.606240 11191 raft_consensus.cc:504] T a5e65ee009cc4291a1dad3f1e23bfd45 P 76c69bf8312c409ba42ff602b95cb83a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50461a7d5f5542f58c69ce683bc81deb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41161 } } peers { permanent_uuid: "d6929b0fa8784b10a4c6584a7be75879" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33433 } } peers { permanent_uuid: "76c69bf8312c409ba42ff602b95cb83a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33557 } }
I20260929 21:00:51.607856 11191 leader_election.cc:288] T a5e65ee009cc4291a1dad3f1e23bfd45 P 76c69bf8312c409ba42ff602b95cb83a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50461a7d5f5542f58c69ce683bc81deb (127.0.97.2:41161), d6929b0fa8784b10a4c6584a7be75879 (127.0.97.1:33433)
I20260929 21:00:51.614351 11169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:51.608203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41161 (local address 127.0.97.3:58049)
0929 21:00:51.608425 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:51.608446 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:51.608583 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:51.608954 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:51.608958 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:51.608983 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:00:51.609220 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:51.609235 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:51.610541 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:51.610545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:51.612334 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:51.612348 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:51.613702 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:51.613708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:51.613898 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:51.613930 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 21:00:51.614035 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 21:00:51.614124 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":49}
I20260929 21:00:51.614476 11180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:51.608326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58049 (local address 127.0.97.2:41161)
0929 21:00:51.608486 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:51.608491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:51.608509 (+ 18us) server_negotiation.cc:407] Connection header received
0929 21:00:51.608731 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:51.608736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:51.608810 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:51.608945 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:00:51.609412 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:51.610403 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:51.612492 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:51.613549 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:51.613676 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:51.613777 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:51.614071 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 21:00:51.614199 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 21:00:51.615136 11184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:51.608792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60871 (local address 127.0.97.1:33433)
0929 21:00:51.609043 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:51.609052 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:51.609592 (+ 540us) server_negotiation.cc:407] Connection header received
0929 21:00:51.609692 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:51.609698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:51.609770 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:51.609869 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:00:51.610289 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:51.611051 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:51.613102 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:51.614237 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:51.614367 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:51.614456 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:51.614784 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 21:00:51.614904 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 21:00:51.615234 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:51.608661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33433 (local address 127.0.97.3:60871)
0929 21:00:51.609492 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0929 21:00:51.609509 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 21:00:51.609595 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:00:51.609870 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:00:51.609872 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:00:51.609885 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:00:51.610150 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:51.610157 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:51.611202 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:51.611207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:51.612963 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:00:51.612973 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:00:51.614450 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:00:51.614458 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:00:51.614637 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:51.614677 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 21:00:51.614762 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 21:00:51.614878 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"mutex_wait_us":1,"thread_start_us":526,"threads_started":1}
I20260929 21:00:51.615495 10989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e65ee009cc4291a1dad3f1e23bfd45" candidate_uuid: "76c69bf8312c409ba42ff602b95cb83a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50461a7d5f5542f58c69ce683bc81deb" is_pre_election: true
I20260929 21:00:51.615969 10989 raft_consensus.cc:2454] T a5e65ee009cc4291a1dad3f1e23bfd45 P 50461a7d5f5542f58c69ce683bc81deb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c69bf8312c409ba42ff602b95cb83a in term 0.
I20260929 21:00:51.616071 10855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e65ee009cc4291a1dad3f1e23bfd45" candidate_uuid: "76c69bf8312c409ba42ff602b95cb83a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6929b0fa8784b10a4c6584a7be75879" is_pre_election: true
I20260929 21:00:51.616580 10855 raft_consensus.cc:2454] T a5e65ee009cc4291a1dad3f1e23bfd45 P d6929b0fa8784b10a4c6584a7be75879 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c69bf8312c409ba42ff602b95cb83a in term 0.
I20260929 21:00:51.616621 11057 leader_election.cc:302] T a5e65ee009cc4291a1dad3f1e23bfd45 P 76c69bf8312c409ba42ff602b95cb83a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50461a7d5f5542f58c69ce683bc81deb, 76c69bf8312c409ba42ff602b95cb83a; no voters:
I20260929 21:00:51.617082 11191 raft_consensus.cc:2781] T a5e65ee009cc4291a1dad3f1e23bfd45 P 76c69bf8312c409ba42ff602b95cb83a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 21:00:51.617264 11191 raft_consensus.cc:482] T a5e65ee009cc4291a1dad3f1e23bfd45 P 76c69bf8312c409ba42ff602b95cb83a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 21:00:51.617385 11191 raft_consensus.cc:3037] T a5e65ee009cc4291a1dad3f1e23bfd45 P 76c69bf8312c409ba42ff602b95cb83a [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:00:51.620008 11191 raft_consensus.cc:504] T a5e65ee009cc4291a1dad3f1e23bfd45 P 76c69bf8312c409ba42ff602b95cb83a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50461a7d5f5542f58c69ce683bc81deb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41161 } } peers { permanent_uuid: "d6929b0fa8784b10a4c6584a7be75879" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33433 } } peers { permanent_uuid: "76c69bf8312c409ba42ff602b95cb83a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33557 } }
I20260929 21:00:51.620523 11191 leader_election.cc:288] T a5e65ee009cc4291a1dad3f1e23bfd45 P 76c69bf8312c409ba42ff602b95cb83a [CANDIDATE]: Term 1 election: Requested vote from peers 50461a7d5f5542f58c69ce683bc81deb (127.0.97.2:41161), d6929b0fa8784b10a4c6584a7be75879 (127.0.97.1:33433)
I20260929 21:00:51.620960 10989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e65ee009cc4291a1dad3f1e23bfd45" candidate_uuid: "76c69bf8312c409ba42ff602b95cb83a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50461a7d5f5542f58c69ce683bc81deb"
I20260929 21:00:51.621138 10989 raft_consensus.cc:3037] T a5e65ee009cc4291a1dad3f1e23bfd45 P 50461a7d5f5542f58c69ce683bc81deb [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:00:51.621135 10855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e65ee009cc4291a1dad3f1e23bfd45" candidate_uuid: "76c69bf8312c409ba42ff602b95cb83a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6929b0fa8784b10a4c6584a7be75879"
I20260929 21:00:51.621356 10855 raft_consensus.cc:3037] T a5e65ee009cc4291a1dad3f1e23bfd45 P d6929b0fa8784b10a4c6584a7be75879 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:00:51.623468 10989 raft_consensus.cc:2454] T a5e65ee009cc4291a1dad3f1e23bfd45 P 50461a7d5f5542f58c69ce683bc81deb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c69bf8312c409ba42ff602b95cb83a in term 1.
I20260929 21:00:51.623669 10855 raft_consensus.cc:2454] T a5e65ee009cc4291a1dad3f1e23bfd45 P d6929b0fa8784b10a4c6584a7be75879 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c69bf8312c409ba42ff602b95cb83a in term 1.
I20260929 21:00:51.623845 11057 leader_election.cc:302] T a5e65ee009cc4291a1dad3f1e23bfd45 P 76c69bf8312c409ba42ff602b95cb83a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50461a7d5f5542f58c69ce683bc81deb, 76c69bf8312c409ba42ff602b95cb83a; no voters:
I20260929 21:00:51.624171 11191 raft_consensus.cc:2781] T a5e65ee009cc4291a1dad3f1e23bfd45 P 76c69bf8312c409ba42ff602b95cb83a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:00:51.624610 11191 raft_consensus.cc:686] T a5e65ee009cc4291a1dad3f1e23bfd45 P 76c69bf8312c409ba42ff602b95cb83a [term 1 LEADER]: Becoming Leader. State: Replica: 76c69bf8312c409ba42ff602b95cb83a, State: Running, Role: LEADER
I20260929 21:00:51.625129 11191 consensus_queue.cc:237] T a5e65ee009cc4291a1dad3f1e23bfd45 P 76c69bf8312c409ba42ff602b95cb83a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50461a7d5f5542f58c69ce683bc81deb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41161 } } peers { permanent_uuid: "d6929b0fa8784b10a4c6584a7be75879" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33433 } } peers { permanent_uuid: "76c69bf8312c409ba42ff602b95cb83a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33557 } }
I20260929 21:00:51.629526 10710 catalog_manager.cc:5495] T a5e65ee009cc4291a1dad3f1e23bfd45 P 76c69bf8312c409ba42ff602b95cb83a reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c69bf8312c409ba42ff602b95cb83a (127.0.97.3). New cstate: current_term: 1 leader_uuid: "76c69bf8312c409ba42ff602b95cb83a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50461a7d5f5542f58c69ce683bc81deb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6929b0fa8784b10a4c6584a7be75879" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76c69bf8312c409ba42ff602b95cb83a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33557 } health_report { overall_health: HEALTHY } } }
W20260929 21:00:51.647608 11168 tablet.cc:2369] T a5e65ee009cc4291a1dad3f1e23bfd45 P 76c69bf8312c409ba42ff602b95cb83a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 21:00:51.656136 10901 tablet.cc:2369] T a5e65ee009cc4291a1dad3f1e23bfd45 P d6929b0fa8784b10a4c6584a7be75879: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 21:00:51.665786 11035 tablet.cc:2369] T a5e65ee009cc4291a1dad3f1e23bfd45 P 50461a7d5f5542f58c69ce683bc81deb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 21:00:51.668592 11182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:51.663245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49740 (local address 127.0.97.3:33557)
0929 21:00:51.663420 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:51.663425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:51.663551 (+ 126us) server_negotiation.cc:407] Connection header received
0929 21:00:51.663758 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:51.663766 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:51.663859 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:51.663988 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:00:51.664312 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:51.665073 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:51.666811 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:51.667188 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:51.667278 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:51.667685 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:00:51.668165 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:00:51.668276 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:51.668337 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 21:00:51.668399 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 21:00:51.676136 10855 raft_consensus.cc:1261] T a5e65ee009cc4291a1dad3f1e23bfd45 P d6929b0fa8784b10a4c6584a7be75879 [term 1 FOLLOWER]: Refusing update from remote peer 76c69bf8312c409ba42ff602b95cb83a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:00:51.676222 10989 raft_consensus.cc:1261] T a5e65ee009cc4291a1dad3f1e23bfd45 P 50461a7d5f5542f58c69ce683bc81deb [term 1 FOLLOWER]: Refusing update from remote peer 76c69bf8312c409ba42ff602b95cb83a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:00:51.677141 11195 consensus_queue.cc:1035] T a5e65ee009cc4291a1dad3f1e23bfd45 P 76c69bf8312c409ba42ff602b95cb83a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6929b0fa8784b10a4c6584a7be75879" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:00:51.677513 11191 consensus_queue.cc:1035] T a5e65ee009cc4291a1dad3f1e23bfd45 P 76c69bf8312c409ba42ff602b95cb83a [LEADER]: Connected to new peer: Peer: permanent_uuid: "50461a7d5f5542f58c69ce683bc81deb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:00:51.687844 11198 mvcc.cc:205] Tried to move back new op lower bound from 7334771309256646656 to 7334771309060345856. Current Snapshot: MvccSnapshot[applied={T|T < 7334771309256646656}]
I20260929 21:00:51.701491 10710 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 21:00:51.701748 10710 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 21:00:51.704967 10710 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 69967fdb7413412281dfd30378ada995: Sending DeleteTablet for 3 replicas of tablet a5e65ee009cc4291a1dad3f1e23bfd45
I20260929 21:00:51.706154 10835 tablet_service.cc:1514] Processing DeleteTablet for tablet a5e65ee009cc4291a1dad3f1e23bfd45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:00:51 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60062
I20260929 21:00:51.706259 11102 tablet_service.cc:1514] Processing DeleteTablet for tablet a5e65ee009cc4291a1dad3f1e23bfd45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:00:51 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49736
I20260929 21:00:51.706418 10969 tablet_service.cc:1514] Processing DeleteTablet for tablet a5e65ee009cc4291a1dad3f1e23bfd45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:00:51 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48040
I20260929 21:00:51.708179 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 10772
I20260929 21:00:51.721733 11209 tablet_replica.cc:330] T a5e65ee009cc4291a1dad3f1e23bfd45 P 76c69bf8312c409ba42ff602b95cb83a: stopping tablet replica
I20260929 21:00:51.722249 11209 raft_consensus.cc:2229] T a5e65ee009cc4291a1dad3f1e23bfd45 P 76c69bf8312c409ba42ff602b95cb83a [term 1 LEADER]: Raft consensus shutting down.
I20260929 21:00:51.722999 11209 raft_consensus.cc:2258] T a5e65ee009cc4291a1dad3f1e23bfd45 P 76c69bf8312c409ba42ff602b95cb83a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:00:51.725517 11209 ts_tablet_manager.cc:1905] T a5e65ee009cc4291a1dad3f1e23bfd45 P 76c69bf8312c409ba42ff602b95cb83a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:00:51.734557 11209 ts_tablet_manager.cc:1918] T a5e65ee009cc4291a1dad3f1e23bfd45 P 76c69bf8312c409ba42ff602b95cb83a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:00:51.734767 11209 log.cc:1196] T a5e65ee009cc4291a1dad3f1e23bfd45 P 76c69bf8312c409ba42ff602b95cb83a: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-2/wal/wals/a5e65ee009cc4291a1dad3f1e23bfd45
I20260929 21:00:51.735376 11209 ts_tablet_manager.cc:1939] T a5e65ee009cc4291a1dad3f1e23bfd45 P 76c69bf8312c409ba42ff602b95cb83a: Deleting consensus metadata
I20260929 21:00:51.738091 10696 catalog_manager.cc:4841] TS 76c69bf8312c409ba42ff602b95cb83a (127.0.97.3:33557): tablet a5e65ee009cc4291a1dad3f1e23bfd45 (table test-table [id=60db4c98ab8348e1b4258abb1c6fc1b3]) successfully deleted
I20260929 21:00:51.739979 11208 tablet_replica.cc:330] T a5e65ee009cc4291a1dad3f1e23bfd45 P d6929b0fa8784b10a4c6584a7be75879: stopping tablet replica
I20260929 21:00:51.740621 11208 raft_consensus.cc:2229] T a5e65ee009cc4291a1dad3f1e23bfd45 P d6929b0fa8784b10a4c6584a7be75879 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:00:51.741302 11208 raft_consensus.cc:2258] T a5e65ee009cc4291a1dad3f1e23bfd45 P d6929b0fa8784b10a4c6584a7be75879 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:00:51.743568 11208 ts_tablet_manager.cc:1905] T a5e65ee009cc4291a1dad3f1e23bfd45 P d6929b0fa8784b10a4c6584a7be75879: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:00:51.743947 11210 tablet_replica.cc:330] T a5e65ee009cc4291a1dad3f1e23bfd45 P 50461a7d5f5542f58c69ce683bc81deb: stopping tablet replica
Sep 29 21:00:51 dist-test-slave-1dms krb5kdc[10666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 21:00:51.744405 11210 raft_consensus.cc:2229] T a5e65ee009cc4291a1dad3f1e23bfd45 P 50461a7d5f5542f58c69ce683bc81deb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:00:51.744906 11210 raft_consensus.cc:2258] T a5e65ee009cc4291a1dad3f1e23bfd45 P 50461a7d5f5542f58c69ce683bc81deb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:00:51.747123 11210 ts_tablet_manager.cc:1905] T a5e65ee009cc4291a1dad3f1e23bfd45 P 50461a7d5f5542f58c69ce683bc81deb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:00:51.750736 11208 ts_tablet_manager.cc:1918] T a5e65ee009cc4291a1dad3f1e23bfd45 P d6929b0fa8784b10a4c6584a7be75879: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:00:51.750943 11208 log.cc:1196] T a5e65ee009cc4291a1dad3f1e23bfd45 P d6929b0fa8784b10a4c6584a7be75879: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-0/wal/wals/a5e65ee009cc4291a1dad3f1e23bfd45
I20260929 21:00:51.751612 11208 ts_tablet_manager.cc:1939] T a5e65ee009cc4291a1dad3f1e23bfd45 P d6929b0fa8784b10a4c6584a7be75879: Deleting consensus metadata
I20260929 21:00:51.753440 10697 catalog_manager.cc:4841] TS d6929b0fa8784b10a4c6584a7be75879 (127.0.97.1:33433): tablet a5e65ee009cc4291a1dad3f1e23bfd45 (table test-table [id=60db4c98ab8348e1b4258abb1c6fc1b3]) successfully deleted
I20260929 21:00:51.753760 11184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:51.709843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60064 (local address 127.0.97.1:33433)
0929 21:00:51.710083 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:51.710092 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:51.734405 (+ 24313us) server_negotiation.cc:407] Connection header received
0929 21:00:51.735067 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:51.735076 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:51.735183 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:51.735262 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:51.737058 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:51.737843 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:51.739530 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:51.740228 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:51.740340 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:51.745966 (+ 5626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:51.745981 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:51.745998 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:51.746042 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:51.750076 (+ 4034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:51.751015 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:51.751023 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:51.751027 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:51.751180 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:51.751728 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:51.751734 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:51.751737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:51.752177 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:51.752524 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:51.753085 (+ 561us) server_negotiation.cc:299] Negotiation successful
0929 21:00:51.753316 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 21:00:51.754225 11210 ts_tablet_manager.cc:1918] T a5e65ee009cc4291a1dad3f1e23bfd45 P 50461a7d5f5542f58c69ce683bc81deb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:00:51.754380 11210 log.cc:1196] T a5e65ee009cc4291a1dad3f1e23bfd45 P 50461a7d5f5542f58c69ce683bc81deb: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790715556572573-388-0/minicluster-data/ts-1/wal/wals/a5e65ee009cc4291a1dad3f1e23bfd45
I20260929 21:00:51.754911 11210 ts_tablet_manager.cc:1939] T a5e65ee009cc4291a1dad3f1e23bfd45 P 50461a7d5f5542f58c69ce683bc81deb: Deleting consensus metadata
I20260929 21:00:51.755169 10895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:00:51.756938 10697 catalog_manager.cc:4841] TS 50461a7d5f5542f58c69ce683bc81deb (127.0.97.2:41161): tablet a5e65ee009cc4291a1dad3f1e23bfd45 (table test-table [id=60db4c98ab8348e1b4258abb1c6fc1b3]) successfully deleted
I20260929 21:00:51.878010 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 10772
I20260929 21:00:51.900875 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 10906
Sep 29 21:00:51 dist-test-slave-1dms krb5kdc[10666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 21:00:51.916692 11180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:51.901746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48048 (local address 127.0.97.2:41161)
0929 21:00:51.901947 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:51.901961 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:51.901981 (+ 20us) server_negotiation.cc:407] Connection header received
0929 21:00:51.902113 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:51.902118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:51.902187 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:51.902291 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:51.903450 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:51.904176 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:51.905435 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:51.905838 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:51.905985 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:51.910030 (+ 4045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:51.910044 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:51.910063 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:51.910093 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:51.913580 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:51.914332 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:51.914341 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:51.914348 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:51.914509 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:51.914903 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:51.914908 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:51.914912 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:51.915350 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:51.915607 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:51.915942 (+ 335us) server_negotiation.cc:299] Negotiation successful
0929 21:00:51.916182 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 21:00:51.917474 11029 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:00:52.005424 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 10906
I20260929 21:00:52.029805 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 11039
Sep 29 21:00:52 dist-test-slave-1dms krb5kdc[10666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 21:00:52.047364 11182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:00:52.030649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49746 (local address 127.0.97.3:33557)
0929 21:00:52.030819 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 21:00:52.030824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:00:52.030843 (+ 19us) server_negotiation.cc:407] Connection header received
0929 21:00:52.031025 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:00:52.031029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:00:52.031086 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:00:52.031166 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:00:52.032931 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:52.033743 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:52.035505 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:00:52.035933 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:00:52.036024 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:00:52.041325 (+ 5301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:00:52.041337 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:00:52.041352 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:00:52.041382 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:00:52.044436 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:52.045177 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:52.045184 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:52.045187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:52.045340 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:00:52.045723 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:00:52.045731 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:00:52.045734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:00:52.046060 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:00:52.046402 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0929 21:00:52.046697 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 21:00:52.046912 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 21:00:52.048108 11162 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:00:52.168779 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 11039
I20260929 21:00:52.190002 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 10679
I20260929 21:00:52.191145 10741 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:00:52.297444 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 10679
2026-09-29T21:00:52Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4238 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 21:00:52 dist-test-slave-1dms krb5kdc[11218](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 21:00:52 dist-test-slave-1dms krb5kdc[11218](info): set up 2 sockets
Sep 29 21:00:52 dist-test-slave-1dms krb5kdc[11218](info): commencing operation
krb5kdc: starting...
W20260929 21:00:53.401809 388 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 21:00:53 dist-test-slave-1dms krb5kdc[11218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T21:00:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:00:53Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260929 21:00:53.732054 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskdDfHCq/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/postgres -l logfile start
2026-09-29 21:00:56.258 UTC [11248] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 21:00:56.258 UTC [11248] LOG: listening on IPv4 address "127.0.97.20", port 43765
2026-09-29 21:00:56.262 UTC [11248] LOG: listening on Unix socket "/tmp/.s.PGSQL.43765"
2026-09-29 21:00:56.287 UTC [11251] LOG: database system was shut down at 2026-09-29 21:00:54 UTC
2026-09-29 21:00:56.290 UTC [11248] LOG: database system is ready to accept connections
I20260929 21:00:57.265434 388 mini_postgres.cc:96] Postgres bound to 43765
2026-09-29 21:00:57.283 UTC [11259] FATAL: database "slave" does not exist
127.0.97.20:43765 - accepting connections
I20260929 21:00:57.284591 388 mini_ranger.cc:161] Starting Ranger...
I20260929 21:00:57.311453 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 21:00:57.811597 388 mini_ranger.cc:87] Created ranger Postgres database
I20260929 21:00:57.811735 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ranger-admin
2026-09-29 21:00:58,111 [I] DB FLAVOR :POSTGRES
2026-09-29 21:00:58,111 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 21:00:58,111 [I] Checking connection..
2026-09-29 21:00:58,111 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 21:00:58,361 [I] Checking connection passed.
2026-09-29 21:00:58,361 [I] --------- Verifying version history table ---------
2026-09-29 21:00:58,361 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 21:00:58,605 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 21:00:58,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 21:00:58,852 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 21:00:58,853 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 21:00:58,853 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43765/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 21:00:59.106 UTC [11323] WARNING: there is no transaction in progress
2026-09-29 21:00:59,112 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 21:00:59,112 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 21:00:59,376 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 21:00:59,376 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 21:00:59,376 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 21:00:59,631 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 21:00:59,876 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-1dms.c.gcp-upstream.internal','N') ;"
2026-09-29 21:01:00,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 21:01:00,372 [I] Table x_portal_user does not exist in database ranger
2026-09-29 21:01:00,373 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 21:01:00,611 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 21:01:00,611 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 21:01:00,858 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 21:01:01,111 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 21:01:01,111 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43765/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 21:01:02.081 UTC [11431] WARNING: there is no transaction in progress
2026-09-29 21:01:02.096 UTC [11431] WARNING: there is no transaction in progress
2026-09-29 21:01:02.110 UTC [11431] WARNING: there is no transaction in progress
2026-09-29 21:01:02.124 UTC [11431] WARNING: there is no transaction in progress
2026-09-29 21:01:02.142 UTC [11431] WARNING: there is no transaction in progress
2026-09-29 21:01:02.162 UTC [11431] WARNING: there is no transaction in progress
2026-09-29 21:01:02.226 UTC [11431] WARNING: there is no transaction in progress
2026-09-29 21:01:02.233 UTC [11431] WARNING: there is no transaction in progress
2026-09-29 21:01:02.246 UTC [11431] WARNING: there is no transaction in progress
2026-09-29 21:01:02.256 UTC [11431] WARNING: there is no transaction in progress
2026-09-29 21:01:02.266 UTC [11431] WARNING: there is no transaction in progress
2026-09-29 21:01:02.280 UTC [11431] WARNING: there is no transaction in progress
2026-09-29 21:01:02.290 UTC [11431] WARNING: there is no transaction in progress
2026-09-29 21:01:02.296 UTC [11431] WARNING: there is no transaction in progress
2026-09-29 21:01:02.302 UTC [11431] WARNING: there is no transaction in progress
2026-09-29 21:01:02.309 UTC [11431] WARNING: there is no transaction in progress
2026-09-29 21:01:02,946 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 21:01:02,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 21:01:03,220 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 21:01:03,221 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 21:01:03,481 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 21:01:03,481 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 21:01:03,727 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 21:01:03,728 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 21:01:03,968 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 21:01:04,215 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-1dms.c.gcp-upstream.internal';"
2026-09-29 21:01:04,449 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 21:01:04,450 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 21:01:04,688 [I] DB_PATCHES have already been applied
I20260929 21:01:04.694726 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskdDfHCq/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 21:01:04.694826 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260929 21:01:04.708992 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:50329
Sep 29, 2026 9:01:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 9:01:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 9:01:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 9:01:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 9:01:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 9:01:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 9:01:05 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 21:01:05.835587 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.551s user 0.008s sys 0.028s
Sep 29, 2026 9:01:24 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 9:01:24 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 29, 2026 9:01:24 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 9:01:24 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 9:01:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 9:01:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 9:01:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 9:01:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 9:01:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 9:01:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 9:01:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 9:01:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 9:01:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 9:01:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 9:01:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 9:01:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 9:01:24 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 9:01:25 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 21:01:26.212666 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 21:01:26 dist-test-slave-1dms krb5kdc[11218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715686, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 21:01:26.389952 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 21:01:26.410496 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 21:01:27.289294 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 21:01:27.289481 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ranger-kms
I20260929 21:01:27.289626 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:43765
2026-09-29 21:01:27,704 [I] DB FLAVOR :POSTGRES
2026-09-29 21:01:27,704 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 21:01:27,704 [I] Checking connection
2026-09-29 21:01:27,704 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43765/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 21:01:27,931 [I] connection success
2026-09-29 21:01:27,932 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 21:01:27,932 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 21:01:27,932 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43765/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 21:01:28,158 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 21:01:28,158 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 21:01:28,158 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 21:01:28,158 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43765/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 21:01:28,442 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 21:01:28.938053 388 mini_ranger_kms.cc:310] Created kms service
I20260929 21:01:29.139382 388 mini_ranger_kms.cc:326] Created kudu user
I20260929 21:01:29.183563 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 21:01:29 dist-test-slave-1dms krb5kdc[11218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715686, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 21:01:29.592253 388 mini_ranger_kms.cc:384] Added ranger policy
I20260929 21:01:29.592629 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 21:01:29.592706 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260929 21:01:29.613225 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11611
I20260929 21:01:29.613425 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:50411
Sep 29, 2026 9:01:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 9:01:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 9:01:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 9:01:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 9:01:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 9:01:31 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 21:01:31.981027 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.591s user 0.090s sys 0.346s
I20260929 21:01:31.981276 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 21:01:31.981487 388 mini_ranger_kms.cc:401] 127.0.97.20:50411/kms/v1/keys
Sep 29 21:01:32 dist-test-slave-1dms krb5kdc[11218](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790715692, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 21:01:32 dist-test-slave-1dms krb5kdc[11218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715692, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 29 21:01:32 dist-test-slave-1dms krb5kdc[11218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715692, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 29 21:01:35 dist-test-slave-1dms krb5kdc[11218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 21:01:35.109994 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35527
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35527
--ranger_config_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:50411/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:01:35.453629 11662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:01:35.453971 11662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:01:35.454133 11662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:01:35.461807 11662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:01:35.461915 11662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:01:35.461990 11662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:01:35.462479 11662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:01:35.462538 11662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:01:35.472474 11662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:50411/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35527
--ranger_config_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:01:35.473778 11662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:01:35.475064 11662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:01:35.487993 11668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:01:35.487967 11667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:01:35.489153 11670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:01:35.489579 11662 server_base.cc:1034] running on GCE node
I20260929 21:01:35.490375 11662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:01:35.491899 11662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:01:35.493113 11662 hybrid_clock.cc:648] HybridClock initialized: now 1790715695493064 us; error 45 us; skew 500 ppm
I20260929 21:01:35.493461 11662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:01:35 dist-test-slave-1dms krb5kdc[11218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:01:35.496753 11662 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 21:01:35.498354 11662 webserver.cc:460] Webserver started at http://127.0.97.62:46817/ using document root <none> and password file <none>
I20260929 21:01:35.499001 11662 fs_manager.cc:359] Metadata directory not provided
I20260929 21:01:35.499125 11662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:01:35.499430 11662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 21:01:35 dist-test-slave-1dms krb5kdc[11218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 21:01:35.546902 11662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "e41ff16ecd5246499577e3af95b496b8"
format_stamp: "Formatted at 2026-09-29 21:01:35 on dist-test-slave-1dms"
server_key: "753c4e17676f3528f0a1eb461880cadc"
server_key_iv: "f5e55a89b98ba302b95172b4e4ef7b07"
server_key_version: "kuduclusterkey@0"
I20260929 21:01:35.547706 11662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "e41ff16ecd5246499577e3af95b496b8"
format_stamp: "Formatted at 2026-09-29 21:01:35 on dist-test-slave-1dms"
server_key: "753c4e17676f3528f0a1eb461880cadc"
server_key_iv: "f5e55a89b98ba302b95172b4e4ef7b07"
server_key_version: "kuduclusterkey@0"
I20260929 21:01:35.552959 11662 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 21:01:35.573015 11677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:01:35.574605 11662 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260929 21:01:35.574787 11662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "e41ff16ecd5246499577e3af95b496b8"
format_stamp: "Formatted at 2026-09-29 21:01:35 on dist-test-slave-1dms"
server_key: "753c4e17676f3528f0a1eb461880cadc"
server_key_iv: "f5e55a89b98ba302b95172b4e4ef7b07"
server_key_version: "kuduclusterkey@0"
I20260929 21:01:35.575093 11662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:01:35.592492 11662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:01:35.597033 11662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:01:35.597337 11662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:01:35.617130 11662 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35527
I20260929 21:01:35.617148 11728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35527 every 8 connection(s)
I20260929 21:01:35.618592 11662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:01:35.622915 11729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:01:35.626274 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 11662
I20260929 21:01:35.626473 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:01:35.627259 388 external_mini_cluster.cc:1428] Setting key 5f16643d4d451f02da8bc16c32aae0f6
I20260929 21:01:35.637499 11729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e41ff16ecd5246499577e3af95b496b8: Bootstrap starting.
I20260929 21:01:35.642618 11729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e41ff16ecd5246499577e3af95b496b8: Neither blocks nor log segments found. Creating new log.
I20260929 21:01:35.644474 11729 log.cc:824] T 00000000000000000000000000000000 P e41ff16ecd5246499577e3af95b496b8: Log is configured to *not* fsync() on all Append() calls
I20260929 21:01:35.648029 11729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e41ff16ecd5246499577e3af95b496b8: No bootstrap required, opened a new log
I20260929 21:01:35.656728 11729 raft_consensus.cc:348] T 00000000000000000000000000000000 P e41ff16ecd5246499577e3af95b496b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e41ff16ecd5246499577e3af95b496b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35527 } }
I20260929 21:01:35.657238 11729 raft_consensus.cc:374] T 00000000000000000000000000000000 P e41ff16ecd5246499577e3af95b496b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:01:35.657346 11729 raft_consensus.cc:729] T 00000000000000000000000000000000 P e41ff16ecd5246499577e3af95b496b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e41ff16ecd5246499577e3af95b496b8, State: Initialized, Role: FOLLOWER
Sep 29 21:01:35 dist-test-slave-1dms krb5kdc[11218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:01:35.658102 11729 consensus_queue.cc:260] T 00000000000000000000000000000000 P e41ff16ecd5246499577e3af95b496b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e41ff16ecd5246499577e3af95b496b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35527 } }
I20260929 21:01:35.658397 11729 raft_consensus.cc:388] T 00000000000000000000000000000000 P e41ff16ecd5246499577e3af95b496b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:01:35.658571 11729 raft_consensus.cc:482] T 00000000000000000000000000000000 P e41ff16ecd5246499577e3af95b496b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:01:35.658811 11729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e41ff16ecd5246499577e3af95b496b8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:01:35.661931 11729 raft_consensus.cc:504] T 00000000000000000000000000000000 P e41ff16ecd5246499577e3af95b496b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e41ff16ecd5246499577e3af95b496b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35527 } }
I20260929 21:01:35.662477 11729 leader_election.cc:302] T 00000000000000000000000000000000 P e41ff16ecd5246499577e3af95b496b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e41ff16ecd5246499577e3af95b496b8; no voters:
I20260929 21:01:35.663180 11729 leader_election.cc:288] T 00000000000000000000000000000000 P e41ff16ecd5246499577e3af95b496b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 21:01:35.663287 11734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e41ff16ecd5246499577e3af95b496b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:01:35.667371 11734 raft_consensus.cc:686] T 00000000000000000000000000000000 P e41ff16ecd5246499577e3af95b496b8 [term 1 LEADER]: Becoming Leader. State: Replica: e41ff16ecd5246499577e3af95b496b8, State: Running, Role: LEADER
I20260929 21:01:35.668159 11734 consensus_queue.cc:237] T 00000000000000000000000000000000 P e41ff16ecd5246499577e3af95b496b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e41ff16ecd5246499577e3af95b496b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35527 } }
I20260929 21:01:35.669096 11732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:01:35.629453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36178 (local address 127.0.97.62:35527)
0929 21:01:35.641461 (+ 12008us) server_negotiation.cc:206] Beginning negotiation
0929 21:01:35.641481 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 21:01:35.641910 (+ 429us) server_negotiation.cc:407] Connection header received
0929 21:01:35.645547 (+ 3637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:01:35.645571 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:01:35.646020 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:01:35.648084 (+ 2064us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:01:35.648639 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:01:35.650676 (+ 2037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:01:35.652566 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:01:35.653702 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:01:35.653884 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:01:35.658781 (+ 4897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:01:35.658792 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:01:35.658816 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:01:35.658868 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:01:35.663229 (+ 4361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:01:35.664100 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:01:35.664109 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:01:35.664117 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:01:35.664282 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:01:35.664735 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:01:35.664739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:01:35.664742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:01:35.665480 (+ 738us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:01:35.666900 (+ 1420us) server_negotiation.cc:1035] Waiting for connection context
0929 21:01:35.667188 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 21:01:35.667623 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11734,"thread_start_us":11486,"threads_started":1}
I20260929 21:01:35.672906 11729 sys_catalog.cc:564] T 00000000000000000000000000000000 P e41ff16ecd5246499577e3af95b496b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:01:35.677219 11735 sys_catalog.cc:455] T 00000000000000000000000000000000 P e41ff16ecd5246499577e3af95b496b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e41ff16ecd5246499577e3af95b496b8. Latest consensus state: current_term: 1 leader_uuid: "e41ff16ecd5246499577e3af95b496b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e41ff16ecd5246499577e3af95b496b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35527 } } }
I20260929 21:01:35.677635 11735 sys_catalog.cc:458] T 00000000000000000000000000000000 P e41ff16ecd5246499577e3af95b496b8 [sys.catalog]: This master's current role is: LEADER
I20260929 21:01:35.678627 11736 sys_catalog.cc:455] T 00000000000000000000000000000000 P e41ff16ecd5246499577e3af95b496b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e41ff16ecd5246499577e3af95b496b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e41ff16ecd5246499577e3af95b496b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35527 } } }
I20260929 21:01:35.679078 11736 sys_catalog.cc:458] T 00000000000000000000000000000000 P e41ff16ecd5246499577e3af95b496b8 [sys.catalog]: This master's current role is: LEADER
I20260929 21:01:35.682456 11744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:01:35.688946 11744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:01:35.713320 11744 catalog_manager.cc:1347] Generated new cluster ID: 392b78ed8a4b431ca9f2ab3ba19f9b09
I20260929 21:01:35.713447 11744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:01:35.714463 11729 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 21:01:35.747503 11744 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:01:35.749559 11744 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:01:35.758853 11744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e41ff16ecd5246499577e3af95b496b8: Generated new TSK 0
I20260929 21:01:35.759898 11744 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 21:01:36.605175 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:50411/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35527
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 21:01:36 dist-test-slave-1dms krb5kdc[11218](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790715696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 21:01:37.052070 11775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:01:37.052377 11775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:01:37.052587 11775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:01:37.059847 11775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:01:37.060003 11775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:01:37.060187 11775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 21:01:37.070497 11775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:50411/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:01:37.071815 11775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:01:37.073259 11775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:01:37.088178 11786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:01:37.089000 11785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:01:37.094094 11788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:01:37.097491 11775 server_base.cc:1034] running on GCE node
I20260929 21:01:37.098184 11775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:01:37.099182 11775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:01:37.100448 11775 hybrid_clock.cc:648] HybridClock initialized: now 1790715697100383 us; error 54 us; skew 500 ppm
I20260929 21:01:37.100816 11775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:01:37 dist-test-slave-1dms krb5kdc[11218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:01:37.105573 11775 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 21:01:37.109349 11775 webserver.cc:460] Webserver started at http://127.0.97.1:39887/ using document root <none> and password file <none>
I20260929 21:01:37.110311 11775 fs_manager.cc:359] Metadata directory not provided
I20260929 21:01:37.110455 11775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:01:37.110813 11775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 21:01:37 dist-test-slave-1dms krb5kdc[11218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 21:01:37.130156 11775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-0/data/instance:
uuid: "3a5ba23e46894c64a62adfdb2fbe5de2"
format_stamp: "Formatted at 2026-09-29 21:01:37 on dist-test-slave-1dms"
server_key: "26f97fb2a61b2863b868a211a66e5afe"
server_key_iv: "64010a907a9eb08b6e5922753b4eb3b0"
server_key_version: "kuduclusterkey@0"
I20260929 21:01:37.131074 11775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3a5ba23e46894c64a62adfdb2fbe5de2"
format_stamp: "Formatted at 2026-09-29 21:01:37 on dist-test-slave-1dms"
server_key: "26f97fb2a61b2863b868a211a66e5afe"
server_key_iv: "64010a907a9eb08b6e5922753b4eb3b0"
server_key_version: "kuduclusterkey@0"
I20260929 21:01:37.136796 11775 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 21:01:37.169873 11795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:01:37.171896 11775 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.004s sys 0.003s
I20260929 21:01:37.172169 11775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-0/wal
uuid: "3a5ba23e46894c64a62adfdb2fbe5de2"
format_stamp: "Formatted at 2026-09-29 21:01:37 on dist-test-slave-1dms"
server_key: "26f97fb2a61b2863b868a211a66e5afe"
server_key_iv: "64010a907a9eb08b6e5922753b4eb3b0"
server_key_version: "kuduclusterkey@0"
I20260929 21:01:37.172549 11775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:01:37.211254 11775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:01:37.216240 11775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:01:37.216584 11775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:01:37.217870 11775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:01:37.219516 11775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:01:37.219604 11775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:01:37.219756 11775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:01:37.219815 11775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:01:37.296590 11775 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39659
I20260929 21:01:37.296649 11907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39659 every 8 connection(s)
I20260929 21:01:37.298277 11775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
I20260929 21:01:37.303822 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 11775
I20260929 21:01:37.304028 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-0/wal/instance
I20260929 21:01:37.304870 388 external_mini_cluster.cc:1428] Setting key 0cd355988c3102499242883b8c4470d4
Sep 29 21:01:37 dist-test-slave-1dms krb5kdc[11218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 21:01:37.336388 11911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:01:37.305006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36281 (local address 127.0.97.62:35527)
0929 21:01:37.317061 (+ 12055us) server_negotiation.cc:206] Beginning negotiation
0929 21:01:37.317068 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:01:37.317093 (+ 25us) server_negotiation.cc:407] Connection header received
0929 21:01:37.317160 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:01:37.317165 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:01:37.317259 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:01:37.319543 (+ 2284us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:01:37.319726 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:01:37.320926 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:01:37.323469 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:01:37.324100 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:01:37.324223 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:01:37.329293 (+ 5070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:01:37.329300 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:01:37.329305 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:01:37.329344 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:01:37.332926 (+ 3582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:01:37.333897 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:01:37.333902 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:01:37.333906 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:01:37.334064 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:01:37.334609 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:01:37.334614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:01:37.334618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:01:37.334915 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:01:37.335078 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 21:01:37.335789 (+ 711us) server_negotiation.cc:299] Negotiation successful
0929 21:01:37.336018 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11788,"thread_start_us":11828,"threads_started":1}
I20260929 21:01:37.337672 11910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:01:37.304277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35527 (local address 127.0.97.1:36281)
0929 21:01:37.305214 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0929 21:01:37.305263 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 21:01:37.307785 (+ 2522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:01:37.317758 (+ 9973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:01:37.317773 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:01:37.318434 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:01:37.319307 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:01:37.319336 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:01:37.321126 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:01:37.321132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:01:37.323248 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:01:37.323265 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:01:37.324536 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:01:37.324543 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:01:37.324866 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:01:37.325705 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:01:37.325743 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:01:37.329001 (+ 3258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:01:37.333102 (+ 4101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:01:37.333111 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:01:37.333131 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:01:37.333735 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:01:37.334206 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:01:37.334211 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:01:37.334215 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:01:37.334467 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:01:37.335081 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:01:37.335091 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:01:37.335427 (+ 336us) client_negotiation.cc:769] Sending connection context
0929 21:01:37.335718 (+ 291us) client_negotiation.cc:240] Negotiation successful
0929 21:01:37.336256 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":758,"threads_started":1}
I20260929 21:01:37.339416 11908 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35527
I20260929 21:01:37.339798 11908 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 21:01:37.340713 11908 heartbeater.cc:511] Master 127.0.97.62:35527 requested a full tablet report, sending...
I20260929 21:01:37.346289 11694 ts_manager.cc:194] Registered new tserver with Master: 3a5ba23e46894c64a62adfdb2fbe5de2 (127.0.97.1:39659)
I20260929 21:01:37.348488 11694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:36281
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 21:01:37.373211 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:50411/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35527
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 21:01:37 dist-test-slave-1dms krb5kdc[11218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 29 21:01:37 dist-test-slave-1dms krb5kdc[11218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260929 21:01:37.889103 11914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:01:37.889559 11914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:01:37.889739 11914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:01:37.900988 11914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:01:37.901619 11914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:01:37.901854 11914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 21:01:37.917167 11914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:50411/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:01:37.919157 11914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:01:37.920802 11914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:01:37.961364 11922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:01:37.969513 11914 server_base.cc:1034] running on GCE node
W20260929 21:01:37.970120 11921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:01:37.970665 11925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:01:37.971618 11914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:01:37.982983 11914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:01:37.986402 11914 hybrid_clock.cc:648] HybridClock initialized: now 1790715697986323 us; error 89 us; skew 500 ppm
I20260929 21:01:37.986780 11914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:01:37 dist-test-slave-1dms krb5kdc[11218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:01:37.990679 11914 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 21:01:37.994741 11914 webserver.cc:460] Webserver started at http://127.0.97.2:43647/ using document root <none> and password file <none>
I20260929 21:01:37.995729 11914 fs_manager.cc:359] Metadata directory not provided
I20260929 21:01:37.995877 11914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:01:37.996263 11914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 21:01:37 dist-test-slave-1dms krb5kdc[11218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 21:01:38.015283 11914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-1/data/instance:
uuid: "b7ae7abf55a3425c86556b655710c9d3"
format_stamp: "Formatted at 2026-09-29 21:01:38 on dist-test-slave-1dms"
server_key: "64abc2c43ebdee92dfb025a7945fa7a9"
server_key_iv: "7a05924e5f1543893bd5abd78afb4ead"
server_key_version: "kuduclusterkey@0"
I20260929 21:01:38.016680 11914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b7ae7abf55a3425c86556b655710c9d3"
format_stamp: "Formatted at 2026-09-29 21:01:38 on dist-test-slave-1dms"
server_key: "64abc2c43ebdee92dfb025a7945fa7a9"
server_key_iv: "7a05924e5f1543893bd5abd78afb4ead"
server_key_version: "kuduclusterkey@0"
I20260929 21:01:38.032939 11914 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.004s sys 0.012s
I20260929 21:01:38.073499 11931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:01:38.075636 11914 fs_manager.cc:725] Time spent opening block manager: real 0.022s user 0.005s sys 0.008s
I20260929 21:01:38.075888 11914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-1/wal
uuid: "b7ae7abf55a3425c86556b655710c9d3"
format_stamp: "Formatted at 2026-09-29 21:01:38 on dist-test-slave-1dms"
server_key: "64abc2c43ebdee92dfb025a7945fa7a9"
server_key_iv: "7a05924e5f1543893bd5abd78afb4ead"
server_key_version: "kuduclusterkey@0"
I20260929 21:01:38.076253 11914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:01:38.125334 11914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:01:38.131129 11914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:01:38.131563 11914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:01:38.136921 11914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:01:38.145568 11914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:01:38.145742 11914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:01:38.145936 11914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:01:38.146044 11914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:01:38.219380 11914 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36993
I20260929 21:01:38.219398 12050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36993 every 8 connection(s)
I20260929 21:01:38.221104 11914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
I20260929 21:01:38.222963 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 11914
I20260929 21:01:38.223150 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-1/wal/instance
I20260929 21:01:38.223716 388 external_mini_cluster.cc:1428] Setting key 4e81e8ee1497c4b8f59a0f8dbe758d83
Sep 29 21:01:38 dist-test-slave-1dms krb5kdc[11218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:01:38.248003 12053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:01:38.223755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35007 (local address 127.0.97.62:35527)
0929 21:01:38.226313 (+ 2558us) server_negotiation.cc:206] Beginning negotiation
0929 21:01:38.226319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:01:38.226336 (+ 17us) server_negotiation.cc:407] Connection header received
0929 21:01:38.228595 (+ 2259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:01:38.228603 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:01:38.228708 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:01:38.228829 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:01:38.230754 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:01:38.231995 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:01:38.234933 (+ 2938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:01:38.235661 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:01:38.235814 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:01:38.240198 (+ 4384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:01:38.240205 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:01:38.240210 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:01:38.240260 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:01:38.243920 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:01:38.245157 (+ 1237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:01:38.245162 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:01:38.245166 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:01:38.245315 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:01:38.245884 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:01:38.245890 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:01:38.245893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:01:38.246261 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:01:38.246421 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 21:01:38.247366 (+ 945us) server_negotiation.cc:299] Negotiation successful
0929 21:01:38.247616 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2440,"thread_start_us":2346,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 21:01:38.249320 12054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:01:38.224493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35527 (local address 127.0.97.2:35007)
0929 21:01:38.225779 (+ 1286us) negotiation.cc:107] Waiting for socket to connect
0929 21:01:38.225837 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 21:01:38.228254 (+ 2417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:01:38.229127 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:01:38.229138 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:01:38.229741 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:01:38.230559 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:01:38.230584 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:01:38.232160 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:01:38.232166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:01:38.234748 (+ 2582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:01:38.234772 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:01:38.235909 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:01:38.235916 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:01:38.236193 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:01:38.237017 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:01:38.237045 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:01:38.239915 (+ 2870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:01:38.244260 (+ 4345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:01:38.244272 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:01:38.244293 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:01:38.245004 (+ 711us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:01:38.245442 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:01:38.245449 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:01:38.245455 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:01:38.245728 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:01:38.246409 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:01:38.246417 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:01:38.246891 (+ 474us) client_negotiation.cc:769] Sending connection context
0929 21:01:38.247240 (+ 349us) client_negotiation.cc:240] Negotiation successful
0929 21:01:38.247914 (+ 674us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":371,"threads_started":1}
I20260929 21:01:38.250852 12051 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35527
I20260929 21:01:38.251145 12051 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 21:01:38.252007 12051 heartbeater.cc:511] Master 127.0.97.62:35527 requested a full tablet report, sending...
I20260929 21:01:38.253736 11694 ts_manager.cc:194] Registered new tserver with Master: b7ae7abf55a3425c86556b655710c9d3 (127.0.97.2:36993)
I20260929 21:01:38.254830 11694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35007
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 21:01:38.280685 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:50411/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35527
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 21:01:38.355711 11908 heartbeater.cc:503] Master 127.0.97.62:35527 was elected leader, sending a full tablet report...
W20260929 21:01:38.620622 12057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:01:38.621050 12057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:01:38.621206 12057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:01:38.631897 12057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:01:38.632014 12057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:01:38.632129 12057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 21:01:38.641764 12057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:50411/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:01:38.642947 12057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:01:38.644146 12057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:01:38.656992 12063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:01:38.657107 12062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:01:38.657020 12065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:01:38.658159 12057 server_base.cc:1034] running on GCE node
I20260929 21:01:38.658710 12057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:01:38.659600 12057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:01:38.660825 12057 hybrid_clock.cc:648] HybridClock initialized: now 1790715698660773 us; error 73 us; skew 500 ppm
I20260929 21:01:38.661199 12057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:01:38 dist-test-slave-1dms krb5kdc[11218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:01:38.670990 12057 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 21:01:38.672468 12057 webserver.cc:460] Webserver started at http://127.0.97.3:40033/ using document root <none> and password file <none>
I20260929 21:01:38.673240 12057 fs_manager.cc:359] Metadata directory not provided
I20260929 21:01:38.673389 12057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:01:38.673797 12057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 21:01:38 dist-test-slave-1dms krb5kdc[11218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 21:01:38.690686 12057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-2/data/instance:
uuid: "1d9841d85d914e75bae35b8ffb29fcf8"
format_stamp: "Formatted at 2026-09-29 21:01:38 on dist-test-slave-1dms"
server_key: "14f3edde560b8e78ea413f1667a8be12"
server_key_iv: "c7c58346a2efca53e563f36d17bcb74a"
server_key_version: "kuduclusterkey@0"
I20260929 21:01:38.691829 12057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1d9841d85d914e75bae35b8ffb29fcf8"
format_stamp: "Formatted at 2026-09-29 21:01:38 on dist-test-slave-1dms"
server_key: "14f3edde560b8e78ea413f1667a8be12"
server_key_iv: "c7c58346a2efca53e563f36d17bcb74a"
server_key_version: "kuduclusterkey@0"
I20260929 21:01:38.697834 12057 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 21:01:38.711885 12072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:01:38.713502 12057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:01:38.713670 12057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-2/wal
uuid: "1d9841d85d914e75bae35b8ffb29fcf8"
format_stamp: "Formatted at 2026-09-29 21:01:38 on dist-test-slave-1dms"
server_key: "14f3edde560b8e78ea413f1667a8be12"
server_key_iv: "c7c58346a2efca53e563f36d17bcb74a"
server_key_version: "kuduclusterkey@0"
I20260929 21:01:38.713878 12057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:01:38.743494 12057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:01:38.747067 12057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:01:38.747351 12057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:01:38.748256 12057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:01:38.749662 12057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:01:38.749732 12057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:01:38.749890 12057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:01:38.749962 12057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:01:38.788604 12057 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38523
I20260929 21:01:38.788641 12184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38523 every 8 connection(s)
I20260929 21:01:38.789913 12057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
I20260929 21:01:38.793977 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 12057
I20260929 21:01:38.794163 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-2/wal/instance
I20260929 21:01:38.794726 388 external_mini_cluster.cc:1428] Setting key 3ed9c7f47c21a452c06b153c4d829438
Sep 29 21:01:38 dist-test-slave-1dms krb5kdc[11218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:01:38.809945 12187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:01:38.792325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36597 (local address 127.0.97.62:35527)
0929 21:01:38.792799 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0929 21:01:38.792804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:01:38.793700 (+ 896us) server_negotiation.cc:407] Connection header received
0929 21:01:38.796208 (+ 2508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:01:38.796214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:01:38.796309 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:01:38.796428 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:01:38.798092 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:01:38.798790 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:01:38.800279 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:01:38.800667 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:01:38.800755 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:01:38.804195 (+ 3440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:01:38.804201 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:01:38.804204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:01:38.804233 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:01:38.806814 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:01:38.807518 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:01:38.807522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:01:38.807524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:01:38.807637 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:01:38.808075 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:01:38.808080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:01:38.808083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:01:38.808444 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:01:38.808594 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 21:01:38.809373 (+ 779us) server_negotiation.cc:299] Negotiation successful
0929 21:01:38.809561 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":267,"threads_started":1}
I20260929 21:01:38.810845 12188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:01:38.792762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35527 (local address 127.0.97.3:36597)
0929 21:01:38.793492 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0929 21:01:38.793534 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 21:01:38.795814 (+ 2280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:01:38.796829 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:01:38.796842 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:01:38.797294 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:01:38.797937 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:01:38.797957 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:01:38.798963 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:01:38.798966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:01:38.800164 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:01:38.800175 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:01:38.800875 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:01:38.800884 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:01:38.801089 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:01:38.801678 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:01:38.801702 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:01:38.803975 (+ 2273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:01:38.806953 (+ 2978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:01:38.806970 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:01:38.806985 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:01:38.807379 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:01:38.807745 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:01:38.807751 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:01:38.807754 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:01:38.807949 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:01:38.808581 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:01:38.808588 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:01:38.808986 (+ 398us) client_negotiation.cc:769] Sending connection context
0929 21:01:38.809266 (+ 280us) client_negotiation.cc:240] Negotiation successful
0929 21:01:38.809745 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":272,"threads_started":1}
I20260929 21:01:38.812131 12185 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35527
I20260929 21:01:38.812415 12185 heartbeater.cc:464] Registering TS with master...
I20260929 21:01:38.813134 12185 heartbeater.cc:511] Master 127.0.97.62:35527 requested a full tablet report, sending...
I20260929 21:01:38.814419 11694 ts_manager.cc:194] Registered new tserver with Master: 1d9841d85d914e75bae35b8ffb29fcf8 (127.0.97.3:38523)
I20260929 21:01:38.815155 11694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36597
I20260929 21:01:38.822031 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 21:01:38.838469 12187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:01:38.826311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43824 (local address 127.0.97.62:35527)
0929 21:01:38.826504 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 21:01:38.826512 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:01:38.826914 (+ 402us) server_negotiation.cc:407] Connection header received
0929 21:01:38.827072 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:01:38.827076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:01:38.827142 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:01:38.827218 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:01:38.828919 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:01:38.829605 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:01:38.830904 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:01:38.831313 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:01:38.831407 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:01:38.833778 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:01:38.833783 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:01:38.833786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:01:38.833822 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:01:38.836342 (+ 2520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:01:38.836972 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:01:38.836976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:01:38.836979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:01:38.837094 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:01:38.837413 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:01:38.837415 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:01:38.837417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:01:38.837650 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:01:38.837816 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 21:01:38.838099 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 21:01:38.838251 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 21:01:38.842329 11694 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 21:01:38.849432 11694 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 21:01:38.873374 12199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:01:38.862311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42912 (local address 127.0.97.3:38523)
0929 21:01:38.863156 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0929 21:01:38.863169 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 21:01:38.863750 (+ 581us) server_negotiation.cc:407] Connection header received
0929 21:01:38.864116 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:01:38.864126 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:01:38.864388 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:01:38.864549 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:01:38.865643 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:01:38.867635 (+ 1992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:01:38.870399 (+ 2764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:01:38.871823 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:01:38.871914 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:01:38.872151 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 21:01:38.872775 (+ 624us) server_negotiation.cc:299] Negotiation successful
0929 21:01:38.872958 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":506,"threads_started":1}
I20260929 21:01:38.874509 12198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:01:38.862019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38523 (local address 127.0.0.1:42912)
0929 21:01:38.863596 (+ 1577us) negotiation.cc:107] Waiting for socket to connect
0929 21:01:38.863642 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 21:01:38.863842 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:01:38.864934 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:01:38.864941 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:01:38.864993 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:01:38.865425 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:01:38.865439 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:01:38.867806 (+ 2367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:01:38.867811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:01:38.870223 (+ 2412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:01:38.870239 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:01:38.872323 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:01:38.872328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:01:38.872509 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:01:38.872565 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 21:01:38.873928 (+ 1363us) client_negotiation.cc:240] Negotiation successful
0929 21:01:38.874130 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1282,"thread_start_us":1138,"threads_started":1}
I20260929 21:01:38.875638 12200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:01:38.862657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36993 (local address 127.0.0.1:37092)
0929 21:01:38.863680 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0929 21:01:38.863697 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 21:01:38.863842 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:01:38.866125 (+ 2283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:01:38.866130 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:01:38.866154 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:01:38.866951 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:01:38.866962 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:01:38.870044 (+ 3082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:01:38.870051 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:01:38.872097 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:01:38.872113 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:01:38.874884 (+ 2771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:01:38.874890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:01:38.875073 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:01:38.875116 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 21:01:38.875178 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 21:01:38.875318 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":911,"thread_start_us":781,"threads_started":1}
I20260929 21:01:38.876813 12201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:01:38.863126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37092 (local address 127.0.97.2:36993)
0929 21:01:38.864397 (+ 1271us) server_negotiation.cc:206] Beginning negotiation
0929 21:01:38.864406 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:01:38.864433 (+ 27us) server_negotiation.cc:407] Connection header received
0929 21:01:38.864532 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:01:38.864538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:01:38.865917 (+ 1379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:01:38.867134 (+ 1217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:01:38.867328 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:01:38.869893 (+ 2565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:01:38.872293 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:01:38.874662 (+ 2369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:01:38.875657 (+ 995us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:01:38.876264 (+ 607us) server_negotiation.cc:1035] Waiting for connection context
0929 21:01:38.876342 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 21:01:38.876523 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1099,"thread_start_us":1461,"threads_started":1}
I20260929 21:01:38.880241 12120 tablet_service.cc:1467] Processing CreateTablet for tablet 2fb9682d3e004b9083ed156e36532adb (DEFAULT_TABLE table=test-table [id=d28e9e5984b34e1cbc6ff8d5c1340e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:01:38.880272 11986 tablet_service.cc:1467] Processing CreateTablet for tablet 2fb9682d3e004b9083ed156e36532adb (DEFAULT_TABLE table=test-table [id=d28e9e5984b34e1cbc6ff8d5c1340e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:01:38.882040 12120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fb9682d3e004b9083ed156e36532adb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:01:38.882084 11986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fb9682d3e004b9083ed156e36532adb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:01:38.883361 12198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:01:38.864021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39659 (local address 127.0.0.1:57900)
0929 21:01:38.874928 (+ 10907us) negotiation.cc:107] Waiting for socket to connect
0929 21:01:38.874947 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 21:01:38.875049 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:01:38.876319 (+ 1270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:01:38.876324 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:01:38.876352 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:01:38.876550 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:01:38.876558 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:01:38.878421 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:01:38.878425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:01:38.880236 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:01:38.880254 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:01:38.882490 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:01:38.882495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:01:38.882665 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:01:38.882705 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 21:01:38.882883 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 21:01:38.883106 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10815,"thread_start_us":12082,"threads_started":1}
I20260929 21:01:38.884259 12203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:01:38.864390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57900 (local address 127.0.97.1:39659)
0929 21:01:38.875579 (+ 11189us) server_negotiation.cc:206] Beginning negotiation
0929 21:01:38.875587 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:01:38.875619 (+ 32us) server_negotiation.cc:407] Connection header received
0929 21:01:38.875722 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:01:38.875732 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:01:38.876005 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:01:38.876215 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:01:38.876693 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:01:38.878245 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:01:38.880393 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:01:38.882296 (+ 1903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:01:38.882432 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:01:38.882636 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 21:01:38.882935 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 21:01:38.883908 (+ 973us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11006,"thread_start_us":10851,"threads_started":1}
I20260929 21:01:38.887141 11843 tablet_service.cc:1467] Processing CreateTablet for tablet 2fb9682d3e004b9083ed156e36532adb (DEFAULT_TABLE table=test-table [id=d28e9e5984b34e1cbc6ff8d5c1340e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:01:38.888370 11843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fb9682d3e004b9083ed156e36532adb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:01:38.895627 12204 tablet_bootstrap.cc:492] T 2fb9682d3e004b9083ed156e36532adb P 1d9841d85d914e75bae35b8ffb29fcf8: Bootstrap starting.
I20260929 21:01:38.899369 12205 tablet_bootstrap.cc:492] T 2fb9682d3e004b9083ed156e36532adb P b7ae7abf55a3425c86556b655710c9d3: Bootstrap starting.
I20260929 21:01:38.900388 12204 tablet_bootstrap.cc:654] T 2fb9682d3e004b9083ed156e36532adb P 1d9841d85d914e75bae35b8ffb29fcf8: Neither blocks nor log segments found. Creating new log.
I20260929 21:01:38.902419 12204 log.cc:824] T 2fb9682d3e004b9083ed156e36532adb P 1d9841d85d914e75bae35b8ffb29fcf8: Log is configured to *not* fsync() on all Append() calls
I20260929 21:01:38.902470 12205 tablet_bootstrap.cc:654] T 2fb9682d3e004b9083ed156e36532adb P b7ae7abf55a3425c86556b655710c9d3: Neither blocks nor log segments found. Creating new log.
I20260929 21:01:38.903602 12205 log.cc:824] T 2fb9682d3e004b9083ed156e36532adb P b7ae7abf55a3425c86556b655710c9d3: Log is configured to *not* fsync() on all Append() calls
I20260929 21:01:38.903980 12206 tablet_bootstrap.cc:492] T 2fb9682d3e004b9083ed156e36532adb P 3a5ba23e46894c64a62adfdb2fbe5de2: Bootstrap starting.
I20260929 21:01:38.905735 12204 tablet_bootstrap.cc:492] T 2fb9682d3e004b9083ed156e36532adb P 1d9841d85d914e75bae35b8ffb29fcf8: No bootstrap required, opened a new log
I20260929 21:01:38.906014 12204 ts_tablet_manager.cc:1397] T 2fb9682d3e004b9083ed156e36532adb P 1d9841d85d914e75bae35b8ffb29fcf8: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260929 21:01:38.907792 12205 tablet_bootstrap.cc:492] T 2fb9682d3e004b9083ed156e36532adb P b7ae7abf55a3425c86556b655710c9d3: No bootstrap required, opened a new log
I20260929 21:01:38.907945 12206 tablet_bootstrap.cc:654] T 2fb9682d3e004b9083ed156e36532adb P 3a5ba23e46894c64a62adfdb2fbe5de2: Neither blocks nor log segments found. Creating new log.
I20260929 21:01:38.908051 12205 ts_tablet_manager.cc:1397] T 2fb9682d3e004b9083ed156e36532adb P b7ae7abf55a3425c86556b655710c9d3: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260929 21:01:38.909138 12206 log.cc:824] T 2fb9682d3e004b9083ed156e36532adb P 3a5ba23e46894c64a62adfdb2fbe5de2: Log is configured to *not* fsync() on all Append() calls
I20260929 21:01:38.912058 12206 tablet_bootstrap.cc:492] T 2fb9682d3e004b9083ed156e36532adb P 3a5ba23e46894c64a62adfdb2fbe5de2: No bootstrap required, opened a new log
I20260929 21:01:38.912300 12206 ts_tablet_manager.cc:1397] T 2fb9682d3e004b9083ed156e36532adb P 3a5ba23e46894c64a62adfdb2fbe5de2: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 21:01:38.914235 12204 raft_consensus.cc:348] T 2fb9682d3e004b9083ed156e36532adb P 1d9841d85d914e75bae35b8ffb29fcf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9841d85d914e75bae35b8ffb29fcf8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38523 } } peers { permanent_uuid: "3a5ba23e46894c64a62adfdb2fbe5de2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39659 } } peers { permanent_uuid: "b7ae7abf55a3425c86556b655710c9d3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36993 } }
I20260929 21:01:38.914579 12204 raft_consensus.cc:374] T 2fb9682d3e004b9083ed156e36532adb P 1d9841d85d914e75bae35b8ffb29fcf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:01:38.914680 12204 raft_consensus.cc:729] T 2fb9682d3e004b9083ed156e36532adb P 1d9841d85d914e75bae35b8ffb29fcf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d9841d85d914e75bae35b8ffb29fcf8, State: Initialized, Role: FOLLOWER
I20260929 21:01:38.915280 12204 consensus_queue.cc:260] T 2fb9682d3e004b9083ed156e36532adb P 1d9841d85d914e75bae35b8ffb29fcf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9841d85d914e75bae35b8ffb29fcf8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38523 } } peers { permanent_uuid: "3a5ba23e46894c64a62adfdb2fbe5de2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39659 } } peers { permanent_uuid: "b7ae7abf55a3425c86556b655710c9d3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36993 } }
I20260929 21:01:38.916324 12185 heartbeater.cc:503] Master 127.0.97.62:35527 was elected leader, sending a full tablet report...
I20260929 21:01:38.916826 12204 ts_tablet_manager.cc:1428] T 2fb9682d3e004b9083ed156e36532adb P 1d9841d85d914e75bae35b8ffb29fcf8: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
I20260929 21:01:38.918318 12205 raft_consensus.cc:348] T 2fb9682d3e004b9083ed156e36532adb P b7ae7abf55a3425c86556b655710c9d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9841d85d914e75bae35b8ffb29fcf8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38523 } } peers { permanent_uuid: "3a5ba23e46894c64a62adfdb2fbe5de2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39659 } } peers { permanent_uuid: "b7ae7abf55a3425c86556b655710c9d3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36993 } }
I20260929 21:01:38.918725 12205 raft_consensus.cc:374] T 2fb9682d3e004b9083ed156e36532adb P b7ae7abf55a3425c86556b655710c9d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:01:38.918838 12205 raft_consensus.cc:729] T 2fb9682d3e004b9083ed156e36532adb P b7ae7abf55a3425c86556b655710c9d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7ae7abf55a3425c86556b655710c9d3, State: Initialized, Role: FOLLOWER
I20260929 21:01:38.919517 12205 consensus_queue.cc:260] T 2fb9682d3e004b9083ed156e36532adb P b7ae7abf55a3425c86556b655710c9d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9841d85d914e75bae35b8ffb29fcf8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38523 } } peers { permanent_uuid: "3a5ba23e46894c64a62adfdb2fbe5de2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39659 } } peers { permanent_uuid: "b7ae7abf55a3425c86556b655710c9d3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36993 } }
I20260929 21:01:38.922065 12051 heartbeater.cc:503] Master 127.0.97.62:35527 was elected leader, sending a full tablet report...
I20260929 21:01:38.922708 12205 ts_tablet_manager.cc:1428] T 2fb9682d3e004b9083ed156e36532adb P b7ae7abf55a3425c86556b655710c9d3: Time spent starting tablet: real 0.015s user 0.009s sys 0.007s
I20260929 21:01:38.924083 12206 raft_consensus.cc:348] T 2fb9682d3e004b9083ed156e36532adb P 3a5ba23e46894c64a62adfdb2fbe5de2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9841d85d914e75bae35b8ffb29fcf8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38523 } } peers { permanent_uuid: "3a5ba23e46894c64a62adfdb2fbe5de2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39659 } } peers { permanent_uuid: "b7ae7abf55a3425c86556b655710c9d3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36993 } }
I20260929 21:01:38.924543 12206 raft_consensus.cc:374] T 2fb9682d3e004b9083ed156e36532adb P 3a5ba23e46894c64a62adfdb2fbe5de2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:01:38.924641 12206 raft_consensus.cc:729] T 2fb9682d3e004b9083ed156e36532adb P 3a5ba23e46894c64a62adfdb2fbe5de2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a5ba23e46894c64a62adfdb2fbe5de2, State: Initialized, Role: FOLLOWER
I20260929 21:01:38.925253 12206 consensus_queue.cc:260] T 2fb9682d3e004b9083ed156e36532adb P 3a5ba23e46894c64a62adfdb2fbe5de2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9841d85d914e75bae35b8ffb29fcf8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38523 } } peers { permanent_uuid: "3a5ba23e46894c64a62adfdb2fbe5de2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39659 } } peers { permanent_uuid: "b7ae7abf55a3425c86556b655710c9d3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36993 } }
I20260929 21:01:38.926975 12206 ts_tablet_manager.cc:1428] T 2fb9682d3e004b9083ed156e36532adb P 3a5ba23e46894c64a62adfdb2fbe5de2: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
W20260929 21:01:38.973249 12052 tablet.cc:2369] T 2fb9682d3e004b9083ed156e36532adb P b7ae7abf55a3425c86556b655710c9d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 21:01:39.041432 12186 tablet.cc:2369] T 2fb9682d3e004b9083ed156e36532adb P 1d9841d85d914e75bae35b8ffb29fcf8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 21:01:39.055805 12212 raft_consensus.cc:482] T 2fb9682d3e004b9083ed156e36532adb P 3a5ba23e46894c64a62adfdb2fbe5de2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 21:01:39.056136 12212 raft_consensus.cc:504] T 2fb9682d3e004b9083ed156e36532adb P 3a5ba23e46894c64a62adfdb2fbe5de2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9841d85d914e75bae35b8ffb29fcf8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38523 } } peers { permanent_uuid: "3a5ba23e46894c64a62adfdb2fbe5de2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39659 } } peers { permanent_uuid: "b7ae7abf55a3425c86556b655710c9d3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36993 } }
I20260929 21:01:39.057840 12212 leader_election.cc:288] T 2fb9682d3e004b9083ed156e36532adb P 3a5ba23e46894c64a62adfdb2fbe5de2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d9841d85d914e75bae35b8ffb29fcf8 (127.0.97.3:38523), b7ae7abf55a3425c86556b655710c9d3 (127.0.97.2:36993)
I20260929 21:01:39.064622 12199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:01:39.058133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59613 (local address 127.0.97.3:38523)
0929 21:01:39.058348 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 21:01:39.058358 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 21:01:39.059040 (+ 682us) server_negotiation.cc:407] Connection header received
0929 21:01:39.059198 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:01:39.059205 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:01:39.059283 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:01:39.059374 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:01:39.060118 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:01:39.060806 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:01:39.062734 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:01:39.063778 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:01:39.063870 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:01:39.063951 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 21:01:39.064308 (+ 357us) server_negotiation.cc:299] Negotiation successful
0929 21:01:39.064392 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 21:01:39.064893 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:01:39.058133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38523 (local address 127.0.97.1:59613)
0929 21:01:39.058936 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0929 21:01:39.058952 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 21:01:39.059070 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:01:39.059404 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:01:39.059408 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:01:39.059449 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:01:39.059968 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:01:39.059978 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:01:39.060984 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:01:39.060989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:01:39.062592 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:01:39.062603 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:01:39.063980 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:01:39.063986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:01:39.064153 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:01:39.064187 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 21:01:39.064286 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 21:01:39.064422 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":19,"spinlock_wait_cycles":23296,"thread_start_us":470,"threads_started":1}
I20260929 21:01:39.065510 12201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:01:39.058307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39415 (local address 127.0.97.2:36993)
0929 21:01:39.058523 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 21:01:39.058531 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:01:39.059209 (+ 678us) server_negotiation.cc:407] Connection header received
0929 21:01:39.059363 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:01:39.059368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:01:39.059457 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:01:39.059557 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:01:39.060161 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:01:39.061546 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:01:39.063463 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:01:39.064432 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:01:39.064565 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:01:39.064660 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 21:01:39.065138 (+ 478us) server_negotiation.cc:299] Negotiation successful
0929 21:01:39.065269 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 21:01:39.065510 12214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:01:39.058273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36993 (local address 127.0.97.1:39415)
0929 21:01:39.059061 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0929 21:01:39.059078 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 21:01:39.059221 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:01:39.059583 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:01:39.059589 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:01:39.059616 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:01:39.059969 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:01:39.059977 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:01:39.061684 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:01:39.061688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:01:39.063309 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:01:39.063323 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:01:39.064661 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:01:39.064669 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:01:39.064973 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:01:39.065018 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 21:01:39.065109 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 21:01:39.065216 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":1,"thread_start_us":482,"threads_started":1}
I20260929 21:01:39.065992 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9682d3e004b9083ed156e36532adb" candidate_uuid: "3a5ba23e46894c64a62adfdb2fbe5de2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d9841d85d914e75bae35b8ffb29fcf8" is_pre_election: true
I20260929 21:01:39.066505 12140 raft_consensus.cc:2454] T 2fb9682d3e004b9083ed156e36532adb P 1d9841d85d914e75bae35b8ffb29fcf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a5ba23e46894c64a62adfdb2fbe5de2 in term 0.
I20260929 21:01:39.066521 12006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9682d3e004b9083ed156e36532adb" candidate_uuid: "3a5ba23e46894c64a62adfdb2fbe5de2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7ae7abf55a3425c86556b655710c9d3" is_pre_election: true
I20260929 21:01:39.067058 12006 raft_consensus.cc:2454] T 2fb9682d3e004b9083ed156e36532adb P b7ae7abf55a3425c86556b655710c9d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a5ba23e46894c64a62adfdb2fbe5de2 in term 0.
I20260929 21:01:39.067200 11799 leader_election.cc:302] T 2fb9682d3e004b9083ed156e36532adb P 3a5ba23e46894c64a62adfdb2fbe5de2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d9841d85d914e75bae35b8ffb29fcf8, 3a5ba23e46894c64a62adfdb2fbe5de2; no voters:
I20260929 21:01:39.067641 12212 raft_consensus.cc:2781] T 2fb9682d3e004b9083ed156e36532adb P 3a5ba23e46894c64a62adfdb2fbe5de2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 21:01:39.067795 12212 raft_consensus.cc:482] T 2fb9682d3e004b9083ed156e36532adb P 3a5ba23e46894c64a62adfdb2fbe5de2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 21:01:39.067896 12212 raft_consensus.cc:3037] T 2fb9682d3e004b9083ed156e36532adb P 3a5ba23e46894c64a62adfdb2fbe5de2 [term 0 FOLLOWER]: Advancing to term 1
W20260929 21:01:39.069793 11909 tablet.cc:2369] T 2fb9682d3e004b9083ed156e36532adb P 3a5ba23e46894c64a62adfdb2fbe5de2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 21:01:39.070765 12212 raft_consensus.cc:504] T 2fb9682d3e004b9083ed156e36532adb P 3a5ba23e46894c64a62adfdb2fbe5de2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9841d85d914e75bae35b8ffb29fcf8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38523 } } peers { permanent_uuid: "3a5ba23e46894c64a62adfdb2fbe5de2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39659 } } peers { permanent_uuid: "b7ae7abf55a3425c86556b655710c9d3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36993 } }
I20260929 21:01:39.071290 12212 leader_election.cc:288] T 2fb9682d3e004b9083ed156e36532adb P 3a5ba23e46894c64a62adfdb2fbe5de2 [CANDIDATE]: Term 1 election: Requested vote from peers 1d9841d85d914e75bae35b8ffb29fcf8 (127.0.97.3:38523), b7ae7abf55a3425c86556b655710c9d3 (127.0.97.2:36993)
I20260929 21:01:39.071856 12006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9682d3e004b9083ed156e36532adb" candidate_uuid: "3a5ba23e46894c64a62adfdb2fbe5de2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7ae7abf55a3425c86556b655710c9d3"
I20260929 21:01:39.071887 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9682d3e004b9083ed156e36532adb" candidate_uuid: "3a5ba23e46894c64a62adfdb2fbe5de2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d9841d85d914e75bae35b8ffb29fcf8"
I20260929 21:01:39.072073 12140 raft_consensus.cc:3037] T 2fb9682d3e004b9083ed156e36532adb P 1d9841d85d914e75bae35b8ffb29fcf8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:01:39.072073 12006 raft_consensus.cc:3037] T 2fb9682d3e004b9083ed156e36532adb P b7ae7abf55a3425c86556b655710c9d3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:01:39.075675 12006 raft_consensus.cc:2454] T 2fb9682d3e004b9083ed156e36532adb P b7ae7abf55a3425c86556b655710c9d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a5ba23e46894c64a62adfdb2fbe5de2 in term 1.
I20260929 21:01:39.075675 12140 raft_consensus.cc:2454] T 2fb9682d3e004b9083ed156e36532adb P 1d9841d85d914e75bae35b8ffb29fcf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a5ba23e46894c64a62adfdb2fbe5de2 in term 1.
I20260929 21:01:39.076092 11797 leader_election.cc:302] T 2fb9682d3e004b9083ed156e36532adb P 3a5ba23e46894c64a62adfdb2fbe5de2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a5ba23e46894c64a62adfdb2fbe5de2, b7ae7abf55a3425c86556b655710c9d3; no voters:
I20260929 21:01:39.076439 12212 raft_consensus.cc:2781] T 2fb9682d3e004b9083ed156e36532adb P 3a5ba23e46894c64a62adfdb2fbe5de2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:01:39.076984 12212 raft_consensus.cc:686] T 2fb9682d3e004b9083ed156e36532adb P 3a5ba23e46894c64a62adfdb2fbe5de2 [term 1 LEADER]: Becoming Leader. State: Replica: 3a5ba23e46894c64a62adfdb2fbe5de2, State: Running, Role: LEADER
I20260929 21:01:39.077571 12212 consensus_queue.cc:237] T 2fb9682d3e004b9083ed156e36532adb P 3a5ba23e46894c64a62adfdb2fbe5de2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9841d85d914e75bae35b8ffb29fcf8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38523 } } peers { permanent_uuid: "3a5ba23e46894c64a62adfdb2fbe5de2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39659 } } peers { permanent_uuid: "b7ae7abf55a3425c86556b655710c9d3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36993 } }
I20260929 21:01:39.082533 11693 catalog_manager.cc:5495] T 2fb9682d3e004b9083ed156e36532adb P 3a5ba23e46894c64a62adfdb2fbe5de2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a5ba23e46894c64a62adfdb2fbe5de2 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "3a5ba23e46894c64a62adfdb2fbe5de2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9841d85d914e75bae35b8ffb29fcf8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a5ba23e46894c64a62adfdb2fbe5de2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7ae7abf55a3425c86556b655710c9d3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36993 } health_report { overall_health: UNKNOWN } } }
I20260929 21:01:39.109110 12203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:01:39.103729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57906 (local address 127.0.97.1:39659)
0929 21:01:39.103920 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 21:01:39.103927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:01:39.103947 (+ 20us) server_negotiation.cc:407] Connection header received
0929 21:01:39.104207 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:01:39.104211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:01:39.104286 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:01:39.104436 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:01:39.104816 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:01:39.105591 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:01:39.107044 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:01:39.107417 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:01:39.107548 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:01:39.108046 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:01:39.108543 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:01:39.108709 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 21:01:39.108759 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 21:01:39.108832 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 21:01:39.117167 12006 raft_consensus.cc:1261] T 2fb9682d3e004b9083ed156e36532adb P b7ae7abf55a3425c86556b655710c9d3 [term 1 FOLLOWER]: Refusing update from remote peer 3a5ba23e46894c64a62adfdb2fbe5de2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:01:39.117182 12140 raft_consensus.cc:1261] T 2fb9682d3e004b9083ed156e36532adb P 1d9841d85d914e75bae35b8ffb29fcf8 [term 1 FOLLOWER]: Refusing update from remote peer 3a5ba23e46894c64a62adfdb2fbe5de2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:01:39.118112 12215 consensus_queue.cc:1035] T 2fb9682d3e004b9083ed156e36532adb P 3a5ba23e46894c64a62adfdb2fbe5de2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d9841d85d914e75bae35b8ffb29fcf8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:01:39.118460 12212 consensus_queue.cc:1035] T 2fb9682d3e004b9083ed156e36532adb P 3a5ba23e46894c64a62adfdb2fbe5de2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7ae7abf55a3425c86556b655710c9d3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:01:39.144515 11694 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 21:01:39.144793 11694 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 21:01:39.148319 11694 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e41ff16ecd5246499577e3af95b496b8: Sending DeleteTablet for 3 replicas of tablet 2fb9682d3e004b9083ed156e36532adb
I20260929 21:01:39.149520 11843 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fb9682d3e004b9083ed156e36532adb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:01:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57900
I20260929 21:01:39.149528 11986 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fb9682d3e004b9083ed156e36532adb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:01:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37092
I20260929 21:01:39.150247 12120 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fb9682d3e004b9083ed156e36532adb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:01:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42912
I20260929 21:01:39.150391 12229 tablet_replica.cc:330] T 2fb9682d3e004b9083ed156e36532adb P b7ae7abf55a3425c86556b655710c9d3: stopping tablet replica
I20260929 21:01:39.150800 12229 raft_consensus.cc:2229] T 2fb9682d3e004b9083ed156e36532adb P b7ae7abf55a3425c86556b655710c9d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:01:39.151281 12229 raft_consensus.cc:2258] T 2fb9682d3e004b9083ed156e36532adb P b7ae7abf55a3425c86556b655710c9d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:01:39.153184 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 11775
I20260929 21:01:39.153487 12229 ts_tablet_manager.cc:1905] T 2fb9682d3e004b9083ed156e36532adb P b7ae7abf55a3425c86556b655710c9d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:01:39.159715 12228 tablet_replica.cc:330] T 2fb9682d3e004b9083ed156e36532adb P 3a5ba23e46894c64a62adfdb2fbe5de2: stopping tablet replica
I20260929 21:01:39.160140 12228 raft_consensus.cc:2229] T 2fb9682d3e004b9083ed156e36532adb P 3a5ba23e46894c64a62adfdb2fbe5de2 [term 1 LEADER]: Raft consensus shutting down.
I20260929 21:01:39.160926 12228 raft_consensus.cc:2258] T 2fb9682d3e004b9083ed156e36532adb P 3a5ba23e46894c64a62adfdb2fbe5de2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:01:39.162454 12229 ts_tablet_manager.cc:1918] T 2fb9682d3e004b9083ed156e36532adb P b7ae7abf55a3425c86556b655710c9d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:01:39.162604 12229 log.cc:1196] T 2fb9682d3e004b9083ed156e36532adb P b7ae7abf55a3425c86556b655710c9d3: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-1/wal/wals/2fb9682d3e004b9083ed156e36532adb
I20260929 21:01:39.163100 12229 ts_tablet_manager.cc:1939] T 2fb9682d3e004b9083ed156e36532adb P b7ae7abf55a3425c86556b655710c9d3: Deleting consensus metadata
I20260929 21:01:39.164935 11679 catalog_manager.cc:4841] TS b7ae7abf55a3425c86556b655710c9d3 (127.0.97.2:36993): tablet 2fb9682d3e004b9083ed156e36532adb (table test-table [id=d28e9e5984b34e1cbc6ff8d5c1340e43]) successfully deleted
I20260929 21:01:39.165241 12228 ts_tablet_manager.cc:1905] T 2fb9682d3e004b9083ed156e36532adb P 3a5ba23e46894c64a62adfdb2fbe5de2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:01:39.165593 12230 tablet_replica.cc:330] T 2fb9682d3e004b9083ed156e36532adb P 1d9841d85d914e75bae35b8ffb29fcf8: stopping tablet replica
I20260929 21:01:39.166015 12230 raft_consensus.cc:2229] T 2fb9682d3e004b9083ed156e36532adb P 1d9841d85d914e75bae35b8ffb29fcf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:01:39.166510 12230 raft_consensus.cc:2258] T 2fb9682d3e004b9083ed156e36532adb P 1d9841d85d914e75bae35b8ffb29fcf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:01:39.168586 12230 ts_tablet_manager.cc:1905] T 2fb9682d3e004b9083ed156e36532adb P 1d9841d85d914e75bae35b8ffb29fcf8: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 21:01:39 dist-test-slave-1dms krb5kdc[11218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 21:01:39.173467 12228 ts_tablet_manager.cc:1918] T 2fb9682d3e004b9083ed156e36532adb P 3a5ba23e46894c64a62adfdb2fbe5de2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:01:39.173645 12228 log.cc:1196] T 2fb9682d3e004b9083ed156e36532adb P 3a5ba23e46894c64a62adfdb2fbe5de2: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-0/wal/wals/2fb9682d3e004b9083ed156e36532adb
I20260929 21:01:39.174357 12228 ts_tablet_manager.cc:1939] T 2fb9682d3e004b9083ed156e36532adb P 3a5ba23e46894c64a62adfdb2fbe5de2: Deleting consensus metadata
I20260929 21:01:39.175412 12230 ts_tablet_manager.cc:1918] T 2fb9682d3e004b9083ed156e36532adb P 1d9841d85d914e75bae35b8ffb29fcf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:01:39.175568 12230 log.cc:1196] T 2fb9682d3e004b9083ed156e36532adb P 1d9841d85d914e75bae35b8ffb29fcf8: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790715556572573-388-0/minicluster-data/ts-2/wal/wals/2fb9682d3e004b9083ed156e36532adb
I20260929 21:01:39.176038 12230 ts_tablet_manager.cc:1939] T 2fb9682d3e004b9083ed156e36532adb P 1d9841d85d914e75bae35b8ffb29fcf8: Deleting consensus metadata
I20260929 21:01:39.176223 11681 catalog_manager.cc:4841] TS 3a5ba23e46894c64a62adfdb2fbe5de2 (127.0.97.1:39659): tablet 2fb9682d3e004b9083ed156e36532adb (table test-table [id=d28e9e5984b34e1cbc6ff8d5c1340e43]) successfully deleted
I20260929 21:01:39.177728 11681 catalog_manager.cc:4841] TS 1d9841d85d914e75bae35b8ffb29fcf8 (127.0.97.3:38523): tablet 2fb9682d3e004b9083ed156e36532adb (table test-table [id=d28e9e5984b34e1cbc6ff8d5c1340e43]) successfully deleted
I20260929 21:01:39.180955 12203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:01:39.155137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57916 (local address 127.0.97.1:39659)
0929 21:01:39.155354 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 21:01:39.155363 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:01:39.159467 (+ 4104us) server_negotiation.cc:407] Connection header received
0929 21:01:39.160144 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:01:39.160154 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:01:39.160259 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:01:39.160378 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:01:39.162493 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:01:39.163769 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:01:39.166222 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:01:39.166939 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:01:39.167049 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:01:39.173148 (+ 6099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:01:39.173164 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:01:39.173185 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:01:39.173245 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:01:39.177779 (+ 4534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:01:39.178575 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:01:39.178585 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:01:39.178588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:01:39.178772 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:01:39.179130 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:01:39.179135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:01:39.179139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:01:39.179623 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:01:39.179910 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0929 21:01:39.180252 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 21:01:39.180512 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 21:01:39.181739 11903 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:01:39.281338 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 11775
I20260929 21:01:39.299546 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 11914
Sep 29 21:01:39 dist-test-slave-1dms krb5kdc[11218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 21:01:39.314096 12201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:01:39.300399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37098 (local address 127.0.97.2:36993)
0929 21:01:39.300571 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 21:01:39.300576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:01:39.300593 (+ 17us) server_negotiation.cc:407] Connection header received
0929 21:01:39.300703 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:01:39.300707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:01:39.300786 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:01:39.300934 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:01:39.302082 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:01:39.302814 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:01:39.304111 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:01:39.304515 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:01:39.304613 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:01:39.308725 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:01:39.308740 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:01:39.308755 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:01:39.308800 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:01:39.311655 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:01:39.312255 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:01:39.312262 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:01:39.312264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:01:39.312386 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:01:39.312706 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:01:39.312709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:01:39.312711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:01:39.313163 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:01:39.313361 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 21:01:39.313669 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 21:01:39.313822 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 21:01:39.314810 12046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:01:39.404177 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 11914
I20260929 21:01:39.421408 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 12057
Sep 29 21:01:39 dist-test-slave-1dms krb5kdc[11218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 21:01:39.436159 12199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:01:39.422183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42916 (local address 127.0.97.3:38523)
0929 21:01:39.422341 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 21:01:39.422346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:01:39.422363 (+ 17us) server_negotiation.cc:407] Connection header received
0929 21:01:39.422468 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:01:39.422480 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:01:39.422551 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:01:39.422639 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:01:39.423910 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:01:39.424633 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:01:39.425989 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:01:39.426366 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:01:39.426476 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:01:39.430699 (+ 4223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:01:39.430713 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:01:39.430730 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:01:39.430762 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:01:39.433714 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:01:39.434359 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:01:39.434366 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:01:39.434368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:01:39.434506 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:01:39.434870 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:01:39.434875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:01:39.434877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:01:39.435255 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:01:39.435483 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 21:01:39.435738 (+ 255us) server_negotiation.cc:299] Negotiation successful
0929 21:01:39.435889 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 21:01:39.436915 12180 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:01:39.525040 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 12057
I20260929 21:01:39.541105 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 11662
I20260929 21:01:39.542287 11724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:01:39.651934 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 11662
I20260929 21:01:39.672607 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 21:01:40.007567 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 21:01:40.007807 388 mini_ranger.cc:66] Stopping Ranger...
I20260929 21:01:40.352624 388 mini_ranger.cc:68] Stopped Ranger
2026-09-29 21:01:40.353 UTC [11248] LOG: received smart shutdown request
2026-09-29 21:01:40.356 UTC [11248] LOG: background worker "logical replication launcher" (PID 11257) exited with exit code 1
2026-09-29 21:01:40.357 UTC [11252] LOG: shutting down
2026-09-29 21:01:40.532 UTC [11248] LOG: database system is shut down
2026-09-29T21:01:40Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48356 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 21:01:40 dist-test-slave-1dms krb5kdc[12244](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 21:01:40 dist-test-slave-1dms krb5kdc[12244](info): set up 2 sockets
Sep 29 21:01:40 dist-test-slave-1dms krb5kdc[12244](info): commencing operation
krb5kdc: starting...
W20260929 21:01:41.749220 388 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.006s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 21:01:41 dist-test-slave-1dms krb5kdc[12244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T21:01:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:01:41Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260929 21:01:42.060447 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskdDfHCq/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/postgres -l logfile start
2026-09-29 21:01:44.524 UTC [12274] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 21:01:44.524 UTC [12274] LOG: listening on IPv4 address "127.0.97.20", port 43797
2026-09-29 21:01:44.527 UTC [12274] LOG: listening on Unix socket "/tmp/.s.PGSQL.43797"
2026-09-29 21:01:44.549 UTC [12277] LOG: database system was shut down at 2026-09-29 21:01:43 UTC
2026-09-29 21:01:44.552 UTC [12274] LOG: database system is ready to accept connections
I20260929 21:01:45.512742 388 mini_postgres.cc:96] Postgres bound to 43797
2026-09-29 21:01:45.529 UTC [12285] FATAL: database "slave" does not exist
127.0.97.20:43797 - accepting connections
I20260929 21:01:45.530072 388 mini_ranger.cc:161] Starting Ranger...
I20260929 21:01:45.560024 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 21:01:46.049800 388 mini_ranger.cc:87] Created ranger Postgres database
I20260929 21:01:46.049896 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ranger-admin
2026-09-29 21:01:46,315 [I] DB FLAVOR :POSTGRES
2026-09-29 21:01:46,316 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 21:01:46,316 [I] Checking connection..
2026-09-29 21:01:46,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 21:01:46,577 [I] Checking connection passed.
2026-09-29 21:01:46,577 [I] --------- Verifying version history table ---------
2026-09-29 21:01:46,578 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 21:01:46,826 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 21:01:46,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 21:01:47,087 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 21:01:47,087 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 21:01:47,088 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43797/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 21:01:47.340 UTC [12349] WARNING: there is no transaction in progress
2026-09-29 21:01:47,347 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 21:01:47,347 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 21:01:47,583 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 21:01:47,583 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 21:01:47,584 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 21:01:47,812 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 21:01:48,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-1dms.c.gcp-upstream.internal','N') ;"
2026-09-29 21:01:48,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 21:01:48,517 [I] Table x_portal_user does not exist in database ranger
2026-09-29 21:01:48,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 21:01:48,767 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 21:01:48,767 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 21:01:48,995 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 21:01:49,250 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 21:01:49,250 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43797/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 21:01:50.224 UTC [12457] WARNING: there is no transaction in progress
2026-09-29 21:01:50.241 UTC [12457] WARNING: there is no transaction in progress
2026-09-29 21:01:50.257 UTC [12457] WARNING: there is no transaction in progress
2026-09-29 21:01:50.273 UTC [12457] WARNING: there is no transaction in progress
2026-09-29 21:01:50.289 UTC [12457] WARNING: there is no transaction in progress
2026-09-29 21:01:50.309 UTC [12457] WARNING: there is no transaction in progress
2026-09-29 21:01:50.371 UTC [12457] WARNING: there is no transaction in progress
2026-09-29 21:01:50.378 UTC [12457] WARNING: there is no transaction in progress
2026-09-29 21:01:50.392 UTC [12457] WARNING: there is no transaction in progress
2026-09-29 21:01:50.402 UTC [12457] WARNING: there is no transaction in progress
2026-09-29 21:01:50.414 UTC [12457] WARNING: there is no transaction in progress
2026-09-29 21:01:50.428 UTC [12457] WARNING: there is no transaction in progress
2026-09-29 21:01:50.438 UTC [12457] WARNING: there is no transaction in progress
2026-09-29 21:01:50.445 UTC [12457] WARNING: there is no transaction in progress
2026-09-29 21:01:50.452 UTC [12457] WARNING: there is no transaction in progress
2026-09-29 21:01:50.459 UTC [12457] WARNING: there is no transaction in progress
2026-09-29 21:01:51,118 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 21:01:51,119 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 21:01:51,344 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 21:01:51,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 21:01:51,579 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 21:01:51,579 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 21:01:51,821 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 21:01:51,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 21:01:52,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 21:01:52,325 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-1dms.c.gcp-upstream.internal';"
2026-09-29 21:01:52,575 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 21:01:52,575 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 21:01:52,831 [I] DB_PATCHES have already been applied
I20260929 21:01:52.837342 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskdDfHCq/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 21:01:52.837425 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260929 21:01:52.851377 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:48457
Sep 29, 2026 9:01:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 9:01:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 9:01:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 9:01:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 9:01:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 9:01:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 9:01:53 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 21:01:53.984462 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.454s user 0.004s sys 0.036s
Sep 29, 2026 9:02:13 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 9:02:13 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 29, 2026 9:02:14 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 9:02:14 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 9:02:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 9:02:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 9:02:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 9:02:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 9:02:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 9:02:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 9:02:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 9:02:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 9:02:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 9:02:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 9:02:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 9:02:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 9:02:14 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 9:02:14 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 21:02:15.437561 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 21:02:15 dist-test-slave-1dms krb5kdc[12244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715735, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 21:02:15.598675 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 21:02:15.619150 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 21:02:16.520321 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 21:02:16.520553 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ranger-kms
I20260929 21:02:16.520640 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:43797
2026-09-29 21:02:16,866 [I] DB FLAVOR :POSTGRES
2026-09-29 21:02:16,867 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 21:02:16,867 [I] Checking connection
2026-09-29 21:02:16,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43797/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 21:02:17,115 [I] connection success
2026-09-29 21:02:17,115 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 21:02:17,115 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 21:02:17,116 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43797/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 21:02:17,377 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 21:02:17,377 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 21:02:17,378 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 21:02:17,378 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43797/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 21:02:17,667 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 21:02:18.111963 388 mini_ranger_kms.cc:310] Created kms service
I20260929 21:02:18.303376 388 mini_ranger_kms.cc:326] Created kudu user
I20260929 21:02:18.346221 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 21:02:18 dist-test-slave-1dms krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715735, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 21:02:18.769701 388 mini_ranger_kms.cc:384] Added ranger policy
I20260929 21:02:18.770028 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskdDfHCq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskdDfHCq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 21:02:18.770081 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260929 21:02:18.787207 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12637
I20260929 21:02:18.787335 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:50229
Sep 29, 2026 9:02:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 9:02:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 9:02:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 9:02:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 9:02:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 9:02:20 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 21:02:21.144996 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.546s user 0.072s sys 0.285s
I20260929 21:02:21.145241 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 21:02:21.145401 388 mini_ranger_kms.cc:401] 127.0.97.20:50229/kms/v1/keys
Sep 29 21:02:21 dist-test-slave-1dms krb5kdc[12244](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790715741, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 21:02:21 dist-test-slave-1dms krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715741, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 29 21:02:22 dist-test-slave-1dms krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715741, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 29 21:02:24 dist-test-slave-1dms krb5kdc[12244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 21:02:24.462773 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33225
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33225
--ranger_config_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:50229/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:02:24.792505 12688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:02:24.792863 12688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:02:24.793027 12688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:02:24.800076 12688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:02:24.800223 12688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:02:24.800318 12688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:02:24.800413 12688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:02:24.800489 12688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:02:24.810389 12688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:50229/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33225
--ranger_config_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:02:24.811674 12688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:02:24.813020 12688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:02:24.825057 12693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:24.825088 12694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:24.825559 12696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:02:24.826227 12688 server_base.cc:1034] running on GCE node
I20260929 21:02:24.826938 12688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:02:24.828429 12688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:02:24.829645 12688 hybrid_clock.cc:648] HybridClock initialized: now 1790715744829599 us; error 42 us; skew 500 ppm
I20260929 21:02:24.829982 12688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:02:24 dist-test-slave-1dms krb5kdc[12244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:02:24.833222 12688 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 21:02:24.834700 12688 webserver.cc:460] Webserver started at http://127.0.97.62:41717/ using document root <none> and password file <none>
I20260929 21:02:24.835330 12688 fs_manager.cc:359] Metadata directory not provided
I20260929 21:02:24.835462 12688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:02:24.835819 12688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 21:02:24 dist-test-slave-1dms krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 21:02:24.892236 12688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "10280e46262c4cd192423aa45512c909"
format_stamp: "Formatted at 2026-09-29 21:02:24 on dist-test-slave-1dms"
server_key: "b064b90732897b73ed85d313615b8ee6"
server_key_iv: "6433f48ac330c36d2990855e646748fd"
server_key_version: "kuduclusterkey@0"
I20260929 21:02:24.893323 12688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "10280e46262c4cd192423aa45512c909"
format_stamp: "Formatted at 2026-09-29 21:02:24 on dist-test-slave-1dms"
server_key: "b064b90732897b73ed85d313615b8ee6"
server_key_iv: "6433f48ac330c36d2990855e646748fd"
server_key_version: "kuduclusterkey@0"
I20260929 21:02:24.900233 12688 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 21:02:24.920110 12703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:24.921944 12688 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 21:02:24.922142 12688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "10280e46262c4cd192423aa45512c909"
format_stamp: "Formatted at 2026-09-29 21:02:24 on dist-test-slave-1dms"
server_key: "b064b90732897b73ed85d313615b8ee6"
server_key_iv: "6433f48ac330c36d2990855e646748fd"
server_key_version: "kuduclusterkey@0"
I20260929 21:02:24.922447 12688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:02:24.942123 12688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:02:24.947340 12688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:02:24.947693 12688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:02:24.967213 12688 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33225
I20260929 21:02:24.967216 12754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33225 every 8 connection(s)
I20260929 21:02:24.969012 12688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:02:24.972743 12755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:02:24.978394 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 12688
I20260929 21:02:24.978614 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:02:24.979417 388 external_mini_cluster.cc:1428] Setting key 9a4e932d18a35159c7aff9394b71a4cc
I20260929 21:02:24.985176 12755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909: Bootstrap starting.
I20260929 21:02:24.990085 12755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909: Neither blocks nor log segments found. Creating new log.
I20260929 21:02:24.991719 12755 log.cc:824] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909: Log is configured to *not* fsync() on all Append() calls
I20260929 21:02:24.999477 12755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909: No bootstrap required, opened a new log
I20260929 21:02:25.008931 12755 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10280e46262c4cd192423aa45512c909" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33225 } }
I20260929 21:02:25.009402 12755 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:02:25.009510 12755 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10280e46262c4cd192423aa45512c909, State: Initialized, Role: FOLLOWER
I20260929 21:02:25.010083 12755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10280e46262c4cd192423aa45512c909" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33225 } }
Sep 29 21:02:25 dist-test-slave-1dms krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:02:25.010280 12755 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:02:25.010424 12755 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:02:25.010601 12755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:02:25.012918 12755 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10280e46262c4cd192423aa45512c909" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33225 } }
I20260929 21:02:25.013427 12755 leader_election.cc:302] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10280e46262c4cd192423aa45512c909; no voters:
I20260929 21:02:25.014051 12755 leader_election.cc:288] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 21:02:25.014200 12760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:02:25.015385 12760 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909 [term 1 LEADER]: Becoming Leader. State: Replica: 10280e46262c4cd192423aa45512c909, State: Running, Role: LEADER
I20260929 21:02:25.015954 12760 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10280e46262c4cd192423aa45512c909" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33225 } }
I20260929 21:02:25.016412 12755 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:02:25.019137 12762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10280e46262c4cd192423aa45512c909. Latest consensus state: current_term: 1 leader_uuid: "10280e46262c4cd192423aa45512c909" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10280e46262c4cd192423aa45512c909" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33225 } } }
I20260929 21:02:25.019541 12762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909 [sys.catalog]: This master's current role is: LEADER
I20260929 21:02:25.020115 12761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10280e46262c4cd192423aa45512c909" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10280e46262c4cd192423aa45512c909" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33225 } } }
I20260929 21:02:25.020404 12761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909 [sys.catalog]: This master's current role is: LEADER
I20260929 21:02:25.021953 12758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:24.981609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45738 (local address 127.0.97.62:33225)
0929 21:02:24.996273 (+ 14664us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:24.996289 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:24.996341 (+ 52us) server_negotiation.cc:407] Connection header received
0929 21:02:24.998926 (+ 2585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:24.998958 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:24.999456 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:24.999910 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:25.001587 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:25.003833 (+ 2246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:25.005824 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:25.006517 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:25.006666 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:25.011388 (+ 4722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:25.011396 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:25.011410 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:25.011449 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:25.015363 (+ 3914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:25.016332 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:25.016340 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:25.016348 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:25.016537 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:25.016995 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:25.017002 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:25.017006 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:25.017894 (+ 888us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:25.019405 (+ 1511us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:25.019891 (+ 486us) server_negotiation.cc:299] Negotiation successful
0929 21:02:25.020443 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14407,"thread_start_us":15443,"threads_started":1}
I20260929 21:02:25.032574 12766 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:02:25.039167 12766 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:02:25.039937 12755 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 21:02:25.057840 12766 catalog_manager.cc:1347] Generated new cluster ID: 49eea509c10241adb0c0b76e1d1331a5
I20260929 21:02:25.057998 12766 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:02:25.095995 12766 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:02:25.098099 12766 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:02:25.126300 12766 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909: Generated new TSK 0
I20260929 21:02:25.127547 12766 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 21:02:25.846992 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:50229/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33225
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 21:02:26 dist-test-slave-1dms krb5kdc[12244](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790715746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 21:02:26.272303 12801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:02:26.272740 12801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:02:26.272936 12801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:02:26.280890 12801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:02:26.281054 12801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:02:26.281227 12801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 21:02:26.291805 12801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:50229/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:02:26.293282 12801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:02:26.294579 12801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:02:26.311364 12811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:26.312668 12812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:02:26.318298 12801 server_base.cc:1034] running on GCE node
W20260929 21:02:26.318353 12815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:02:26.319221 12801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:02:26.320461 12801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:02:26.321694 12801 hybrid_clock.cc:648] HybridClock initialized: now 1790715746321629 us; error 62 us; skew 500 ppm
I20260929 21:02:26.322034 12801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:02:26 dist-test-slave-1dms krb5kdc[12244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:02:26.325600 12801 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 21:02:26.328042 12801 webserver.cc:460] Webserver started at http://127.0.97.1:46569/ using document root <none> and password file <none>
I20260929 21:02:26.329164 12801 fs_manager.cc:359] Metadata directory not provided
I20260929 21:02:26.329298 12801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:02:26.329667 12801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 21:02:26 dist-test-slave-1dms krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 21:02:26.352496 12801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-0/data/instance:
uuid: "852489c43e7b45dd90e71259986e0ab2"
format_stamp: "Formatted at 2026-09-29 21:02:26 on dist-test-slave-1dms"
server_key: "b6a0a81ee6835424f0fa0a0a7dce0862"
server_key_iv: "dd0eb1ad816a8997890c00b5f81fa264"
server_key_version: "kuduclusterkey@0"
I20260929 21:02:26.355986 12801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-0/wal/instance:
uuid: "852489c43e7b45dd90e71259986e0ab2"
format_stamp: "Formatted at 2026-09-29 21:02:26 on dist-test-slave-1dms"
server_key: "b6a0a81ee6835424f0fa0a0a7dce0862"
server_key_iv: "dd0eb1ad816a8997890c00b5f81fa264"
server_key_version: "kuduclusterkey@0"
I20260929 21:02:26.369249 12801 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.008s sys 0.004s
I20260929 21:02:26.408524 12821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:26.410800 12801 fs_manager.cc:725] Time spent opening block manager: real 0.022s user 0.014s sys 0.007s
I20260929 21:02:26.411005 12801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-0/wal
uuid: "852489c43e7b45dd90e71259986e0ab2"
format_stamp: "Formatted at 2026-09-29 21:02:26 on dist-test-slave-1dms"
server_key: "b6a0a81ee6835424f0fa0a0a7dce0862"
server_key_iv: "dd0eb1ad816a8997890c00b5f81fa264"
server_key_version: "kuduclusterkey@0"
I20260929 21:02:26.411263 12801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:02:26.461679 12801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:02:26.466735 12801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:02:26.467059 12801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:02:26.468120 12801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:02:26.469718 12801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:02:26.469789 12801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:26.469914 12801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:02:26.469988 12801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:26.558393 12801 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45259
I20260929 21:02:26.558409 12933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45259 every 8 connection(s)
I20260929 21:02:26.559871 12801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
I20260929 21:02:26.560874 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 12801
I20260929 21:02:26.561098 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-0/wal/instance
I20260929 21:02:26.561923 388 external_mini_cluster.cc:1428] Setting key 9c8a8234cca97e0edad0202057e42248
Sep 29 21:02:26 dist-test-slave-1dms krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 21:02:26.633241 12937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:26.564493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55629 (local address 127.0.97.62:33225)
0929 21:02:26.598526 (+ 34033us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:26.598537 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:26.603918 (+ 5381us) server_negotiation.cc:407] Connection header received
0929 21:02:26.604014 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:26.604022 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:26.604143 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:26.604274 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:26.615207 (+ 10933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:26.616480 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:26.619327 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:26.620131 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:26.620246 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:26.624777 (+ 4531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:26.624788 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:26.624796 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:26.624879 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:26.628926 (+ 4047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:26.630006 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:26.630014 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:26.630019 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:26.630211 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:26.630827 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:26.630833 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:26.630837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:26.631225 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:26.631386 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:26.632494 (+ 1108us) server_negotiation.cc:299] Negotiation successful
0929 21:02:26.632763 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33837,"thread_start_us":34069,"threads_started":1}
I20260929 21:02:26.635730 12936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:26.563915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33225 (local address 127.0.97.1:55629)
0929 21:02:26.599481 (+ 35566us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:26.599575 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0929 21:02:26.603285 (+ 3710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:02:26.613383 (+ 10098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:02:26.613398 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:02:26.614112 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:02:26.614960 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:26.614988 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:26.616761 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:26.616769 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:26.619156 (+ 2387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:26.619175 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:26.620606 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:26.620614 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:26.620934 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:26.621756 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:02:26.621786 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:02:26.624469 (+ 2683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:02:26.629092 (+ 4623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:26.629104 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:26.629128 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:26.629835 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:26.630385 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:26.630393 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:26.630398 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:26.630689 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:26.631485 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:02:26.631497 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:02:26.631991 (+ 494us) client_negotiation.cc:769] Sending connection context
0929 21:02:26.633531 (+ 1540us) client_negotiation.cc:240] Negotiation successful
0929 21:02:26.634124 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35082,"thread_start_us":46490,"threads_started":1}
I20260929 21:02:26.638094 12934 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33225
I20260929 21:02:26.638422 12934 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 21:02:26.639303 12934 heartbeater.cc:511] Master 127.0.97.62:33225 requested a full tablet report, sending...
I20260929 21:02:26.641883 12720 ts_manager.cc:194] Registered new tserver with Master: 852489c43e7b45dd90e71259986e0ab2 (127.0.97.1:45259)
I20260929 21:02:26.644105 12720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55629
Sep 29 21:02:26 dist-test-slave-1dms krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab.
W20260929 21:02:26.673926 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.2: real 0.109s user 0.004s sys 0.013s
I20260929 21:02:26.674841 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:50229/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33225
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 21:02:26 dist-test-slave-1dms krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260929 21:02:27.167814 12940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:02:27.168152 12940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:02:27.168329 12940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:02:27.178853 12940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:02:27.179051 12940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:02:27.179281 12940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 21:02:27.193843 12940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:50229/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:02:27.195634 12940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:02:27.197144 12940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:02:27.224035 12947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:27.224732 12948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:02:27.227972 12940 server_base.cc:1034] running on GCE node
W20260929 21:02:27.228296 12950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:02:27.229223 12940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:02:27.234064 12940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:02:27.235363 12940 hybrid_clock.cc:648] HybridClock initialized: now 1790715747235273 us; error 71 us; skew 500 ppm
I20260929 21:02:27.235811 12940 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:02:27 dist-test-slave-1dms krb5kdc[12244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:02:27.239997 12940 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 21:02:27.242823 12940 webserver.cc:460] Webserver started at http://127.0.97.2:35851/ using document root <none> and password file <none>
I20260929 21:02:27.243773 12940 fs_manager.cc:359] Metadata directory not provided
I20260929 21:02:27.243902 12940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:02:27.244228 12940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 21:02:27 dist-test-slave-1dms krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 21:02:27.264583 12940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-1/data/instance:
uuid: "2a378a26b7254ae6be02700ef74ef773"
format_stamp: "Formatted at 2026-09-29 21:02:27 on dist-test-slave-1dms"
server_key: "342d8782b97334f94ec7aff81d28d24f"
server_key_iv: "b901781b323ea3be2c1d819ccfa4531c"
server_key_version: "kuduclusterkey@0"
I20260929 21:02:27.265688 12940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2a378a26b7254ae6be02700ef74ef773"
format_stamp: "Formatted at 2026-09-29 21:02:27 on dist-test-slave-1dms"
server_key: "342d8782b97334f94ec7aff81d28d24f"
server_key_iv: "b901781b323ea3be2c1d819ccfa4531c"
server_key_version: "kuduclusterkey@0"
I20260929 21:02:27.273398 12940 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260929 21:02:27.291822 12964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:27.293390 12940 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:02:27.293589 12940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-1/wal
uuid: "2a378a26b7254ae6be02700ef74ef773"
format_stamp: "Formatted at 2026-09-29 21:02:27 on dist-test-slave-1dms"
server_key: "342d8782b97334f94ec7aff81d28d24f"
server_key_iv: "b901781b323ea3be2c1d819ccfa4531c"
server_key_version: "kuduclusterkey@0"
I20260929 21:02:27.293833 12940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:02:27.323799 12940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:02:27.327948 12940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:02:27.328290 12940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:02:27.329371 12940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:02:27.330760 12940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:02:27.330859 12940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:27.331010 12940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:02:27.331086 12940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:27.364487 12940 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36283
I20260929 21:02:27.364529 13076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36283 every 8 connection(s)
I20260929 21:02:27.365896 12940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
I20260929 21:02:27.372722 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 12940
I20260929 21:02:27.372907 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-1/wal/instance
I20260929 21:02:27.373435 388 external_mini_cluster.cc:1428] Setting key 1e07ada893591ed364ed85d23702f865
Sep 29 21:02:27 dist-test-slave-1dms krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:02:27.386380 13079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:27.368535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39719 (local address 127.0.97.62:33225)
0929 21:02:27.369025 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:27.369029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:27.369993 (+ 964us) server_negotiation.cc:407] Connection header received
0929 21:02:27.371790 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:27.371795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:27.371920 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:27.372039 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:27.373919 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:27.374702 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:27.376301 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:27.376705 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:27.376879 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:27.380469 (+ 3590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:27.380474 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:27.380477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:27.380510 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:27.383281 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:27.383999 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:27.384009 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:27.384012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:27.384135 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:27.384597 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:27.384601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:27.384603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:27.384924 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:27.385078 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:27.385747 (+ 669us) server_negotiation.cc:299] Negotiation successful
0929 21:02:27.385982 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":221,"threads_started":1}
I20260929 21:02:27.387367 13080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:27.369133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33225 (local address 127.0.97.2:39719)
0929 21:02:27.369837 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:27.369874 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 21:02:27.371537 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:02:27.372280 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:02:27.372288 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:02:27.372900 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:02:27.373729 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:27.373754 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:27.374875 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:27.374883 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:27.376148 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:27.376160 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:27.376963 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:27.376970 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:27.377208 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:27.377913 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:02:27.377935 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:02:27.380236 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:02:27.383423 (+ 3187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:27.383432 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:27.383449 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:27.383866 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:27.384244 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:27.384249 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:27.384253 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:27.384460 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:27.385053 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:02:27.385067 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:02:27.385393 (+ 326us) client_negotiation.cc:769] Sending connection context
0929 21:02:27.385655 (+ 262us) client_negotiation.cc:240] Negotiation successful
0929 21:02:27.386202 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":297,"threads_started":1}
I20260929 21:02:27.388988 13077 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33225
I20260929 21:02:27.389292 13077 heartbeater.cc:464] Registering TS with master...
I20260929 21:02:27.389992 13077 heartbeater.cc:511] Master 127.0.97.62:33225 requested a full tablet report, sending...
I20260929 21:02:27.391498 12720 ts_manager.cc:194] Registered new tserver with Master: 2a378a26b7254ae6be02700ef74ef773 (127.0.97.2:36283)
I20260929 21:02:27.392315 12720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39719
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 21:02:27.424733 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:50229/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33225
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 21:02:27.650389 12934 heartbeater.cc:503] Master 127.0.97.62:33225 was elected leader, sending a full tablet report...
W20260929 21:02:27.751595 13083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:02:27.751871 13083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:02:27.752031 13083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:02:27.759410 13083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:02:27.759550 13083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:02:27.759728 13083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 21:02:27.769547 13083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:50229/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:02:27.770813 13083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:02:27.772104 13083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:02:27.784993 13089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:27.785225 13088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:27.785238 13091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:02:27.786098 13083 server_base.cc:1034] running on GCE node
I20260929 21:02:27.786664 13083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:02:27.787539 13083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:02:27.788736 13083 hybrid_clock.cc:648] HybridClock initialized: now 1790715747788704 us; error 33 us; skew 500 ppm
I20260929 21:02:27.789083 13083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:02:27 dist-test-slave-1dms krb5kdc[12244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:02:27.792335 13083 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 21:02:27.793797 13083 webserver.cc:460] Webserver started at http://127.0.97.3:45251/ using document root <none> and password file <none>
I20260929 21:02:27.794441 13083 fs_manager.cc:359] Metadata directory not provided
I20260929 21:02:27.794592 13083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:02:27.794912 13083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 21:02:27 dist-test-slave-1dms krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 21:02:27.809064 13083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-2/data/instance:
uuid: "73eb589056bd4be3a1a282948eaeb4be"
format_stamp: "Formatted at 2026-09-29 21:02:27 on dist-test-slave-1dms"
server_key: "bef8c1b47e33aab4769cc06db631d49f"
server_key_iv: "e6620b32e10d8530d53bdf487204e3db"
server_key_version: "kuduclusterkey@0"
I20260929 21:02:27.809873 13083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-2/wal/instance:
uuid: "73eb589056bd4be3a1a282948eaeb4be"
format_stamp: "Formatted at 2026-09-29 21:02:27 on dist-test-slave-1dms"
server_key: "bef8c1b47e33aab4769cc06db631d49f"
server_key_iv: "e6620b32e10d8530d53bdf487204e3db"
server_key_version: "kuduclusterkey@0"
I20260929 21:02:27.815083 13083 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 21:02:27.831174 13098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:27.832957 13083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:02:27.833156 13083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-2/wal
uuid: "73eb589056bd4be3a1a282948eaeb4be"
format_stamp: "Formatted at 2026-09-29 21:02:27 on dist-test-slave-1dms"
server_key: "bef8c1b47e33aab4769cc06db631d49f"
server_key_iv: "e6620b32e10d8530d53bdf487204e3db"
server_key_version: "kuduclusterkey@0"
I20260929 21:02:27.833400 13083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:02:27.862111 13083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:02:27.866309 13083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:02:27.866600 13083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:02:27.867543 13083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:02:27.868872 13083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:02:27.868989 13083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:27.869118 13083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:02:27.869187 13083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:27.901523 13083 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41101
I20260929 21:02:27.901566 13210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41101 every 8 connection(s)
I20260929 21:02:27.902783 13083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
I20260929 21:02:27.907403 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 13083
I20260929 21:02:27.907598 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-2/wal/instance
I20260929 21:02:27.908176 388 external_mini_cluster.cc:1428] Setting key 94d2eb9e5419809e5cb6ea479c1bfeb5
Sep 29 21:02:27 dist-test-slave-1dms krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:02:27.922771 13213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:27.905289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40503 (local address 127.0.97.62:33225)
0929 21:02:27.905765 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:27.905770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:27.906766 (+ 996us) server_negotiation.cc:407] Connection header received
0929 21:02:27.909433 (+ 2667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:27.909436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:27.909499 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:27.909616 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:27.911206 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:27.911918 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:27.913504 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:27.913916 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:27.914045 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:27.917437 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:27.917442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:27.917446 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:27.917488 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:27.919937 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:27.920560 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:27.920565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:27.920567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:27.920703 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:27.921162 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:27.921167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:27.921169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:27.921410 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:27.921577 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:27.922180 (+ 603us) server_negotiation.cc:299] Negotiation successful
0929 21:02:27.922407 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":258,"threads_started":1}
I20260929 21:02:27.923749 13214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:27.905758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33225 (local address 127.0.97.3:40503)
0929 21:02:27.906600 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:27.906638 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 21:02:27.908960 (+ 2322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:02:27.909842 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:02:27.909851 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:02:27.910292 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:02:27.911051 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:27.911071 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:27.912062 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:27.912067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:27.913370 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:27.913382 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:27.914133 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:27.914140 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:27.914365 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:27.915024 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:02:27.915050 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:02:27.917207 (+ 2157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:02:27.920048 (+ 2841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:27.920055 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:27.920067 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:27.920458 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:27.920790 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:27.920793 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:27.920795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:27.921030 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:27.921534 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:02:27.921542 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:02:27.921845 (+ 303us) client_negotiation.cc:769] Sending connection context
0929 21:02:27.922097 (+ 252us) client_negotiation.cc:240] Negotiation successful
0929 21:02:27.922558 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":457,"threads_started":1}
I20260929 21:02:27.925024 13211 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33225
I20260929 21:02:27.925310 13211 heartbeater.cc:464] Registering TS with master...
I20260929 21:02:27.925980 13211 heartbeater.cc:511] Master 127.0.97.62:33225 requested a full tablet report, sending...
I20260929 21:02:27.927361 12720 ts_manager.cc:194] Registered new tserver with Master: 73eb589056bd4be3a1a282948eaeb4be (127.0.97.3:41101)
I20260929 21:02:27.928198 12720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40503
I20260929 21:02:27.936040 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 21:02:27.938127 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 12801
Sep 29 21:02:27 dist-test-slave-1dms krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 21:02:27.953735 13216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:27.939032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54416 (local address 127.0.97.1:45259)
0929 21:02:27.939669 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:27.939676 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:27.939701 (+ 25us) server_negotiation.cc:407] Connection header received
0929 21:02:27.939758 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:27.939763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:27.939953 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:27.940077 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:27.941551 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:27.942547 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:27.943980 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:27.944492 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:27.944590 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:27.948656 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:27.948674 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:27.948687 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:27.948720 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:27.951328 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:27.951921 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:27.951927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:27.951929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:27.952058 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:27.952365 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:27.952367 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:27.952369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:27.952681 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:27.952961 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:27.953266 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 21:02:27.953454 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":324,"threads_started":1}
I20260929 21:02:27.955034 12929 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:02:28.044380 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 12801
I20260929 21:02:28.061085 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 12940
Sep 29 21:02:28 dist-test-slave-1dms krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 21:02:28.076896 13218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:28.062041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52894 (local address 127.0.97.2:36283)
0929 21:02:28.062567 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:28.062574 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:28.062599 (+ 25us) server_negotiation.cc:407] Connection header received
0929 21:02:28.062677 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:28.062683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:28.062904 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:28.063036 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:28.064242 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:28.065253 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:28.066640 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:28.067173 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:28.067268 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:28.071669 (+ 4401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:28.071680 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:28.071694 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:28.071725 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:28.074567 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:28.075182 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:28.075189 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:28.075191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:28.075305 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:28.075607 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:28.075610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:28.075612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:28.075912 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:28.076140 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:28.076370 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 21:02:28.076555 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":275,"threads_started":1}
I20260929 21:02:28.078130 13072 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:02:28.161131 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 12940
I20260929 21:02:28.177139 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 13083
Sep 29 21:02:28 dist-test-slave-1dms krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 21:02:28.194108 13220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:28.178238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49878 (local address 127.0.97.3:41101)
0929 21:02:28.178840 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:28.178847 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:28.178875 (+ 28us) server_negotiation.cc:407] Connection header received
0929 21:02:28.178932 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:28.178939 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:28.179186 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:28.179313 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:28.180674 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:28.181799 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:28.183134 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:28.183682 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:28.183781 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:28.188129 (+ 4348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:28.188140 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:28.188156 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:28.188192 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:28.191495 (+ 3303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:28.192164 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:28.192172 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:28.192174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:28.192292 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:28.192600 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:28.192603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:28.192605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:28.192996 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:28.193236 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:28.193520 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 21:02:28.193712 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":295,"threads_started":1}
I20260929 21:02:28.195338 13206 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:02:28.280740 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 13083
I20260929 21:02:28.296258 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 12688
I20260929 21:02:28.297467 12750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:02:28.404907 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 12688
I20260929 21:02:28.426717 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33225
--webserver_interface=127.0.97.62
--webserver_port=41717
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33225
--ranger_config_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:50229/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:50229/kms with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:02:28.750918 13223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:02:28.751209 13223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:02:28.751364 13223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:02:28.758791 13223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:02:28.758911 13223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:02:28.759024 13223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:02:28.759131 13223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:02:28.759227 13223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:02:28.769708 13223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:50229/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33225
--ranger_config_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=41717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:02:28.771034 13223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:02:28.772378 13223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:02:28.784403 13228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:28.784330 13231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:28.784332 13229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:02:28.785282 13223 server_base.cc:1034] running on GCE node
I20260929 21:02:28.786000 13223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:02:28.787456 13223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:02:28.788669 13223 hybrid_clock.cc:648] HybridClock initialized: now 1790715748788602 us; error 53 us; skew 500 ppm
I20260929 21:02:28.789064 13223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:02:28 dist-test-slave-1dms krb5kdc[12244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:02:28.797158 13223 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 21:02:28.798645 13223 webserver.cc:460] Webserver started at http://127.0.97.62:41717/ using document root <none> and password file <none>
I20260929 21:02:28.799269 13223 fs_manager.cc:359] Metadata directory not provided
I20260929 21:02:28.799372 13223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 21:02:28 dist-test-slave-1dms krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 21:02:28.856458 13223 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 21:02:28.860041 13239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:28.861692 13223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:02:28.861860 13223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "10280e46262c4cd192423aa45512c909"
format_stamp: "Formatted at 2026-09-29 21:02:24 on dist-test-slave-1dms"
server_key: "b064b90732897b73ed85d313615b8ee6"
server_key_iv: "6433f48ac330c36d2990855e646748fd"
server_key_version: "kuduclusterkey@0"
I20260929 21:02:28.862833 13223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:02:28.892481 13223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:02:28.896636 13223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:02:28.896899 13223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:02:28.916114 13223 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33225
I20260929 21:02:28.916131 13290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33225 every 8 connection(s)
I20260929 21:02:28.917651 13223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:02:28.919695 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 13223
I20260929 21:02:28.920866 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:50229/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:45259
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:33225
--webserver_port=46569
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:50229/kms with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 21:02:28.928521 13291 sys_catalog.cc:263] Verifying existing consensus state
I20260929 21:02:28.932351 13291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909: Bootstrap starting.
I20260929 21:02:28.954540 13291 log.cc:824] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909: Log is configured to *not* fsync() on all Append() calls
I20260929 21:02:28.963699 13291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 21:02:28.964291 13291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909: Bootstrap complete.
I20260929 21:02:28.974393 13291 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10280e46262c4cd192423aa45512c909" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33225 } }
I20260929 21:02:28.975386 13291 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10280e46262c4cd192423aa45512c909, State: Initialized, Role: FOLLOWER
I20260929 21:02:28.976068 13291 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10280e46262c4cd192423aa45512c909" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33225 } }
I20260929 21:02:28.976315 13291 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:02:28.976457 13291 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:02:28.976663 13291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909 [term 1 FOLLOWER]: Advancing to term 2
I20260929 21:02:28.981424 13291 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10280e46262c4cd192423aa45512c909" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33225 } }
I20260929 21:02:28.981953 13291 leader_election.cc:302] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10280e46262c4cd192423aa45512c909; no voters:
I20260929 21:02:28.982671 13291 leader_election.cc:288] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 21:02:28.982829 13295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 21:02:28.983999 13295 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909 [term 2 LEADER]: Becoming Leader. State: Replica: 10280e46262c4cd192423aa45512c909, State: Running, Role: LEADER
I20260929 21:02:28.984660 13295 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10280e46262c4cd192423aa45512c909" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33225 } }
I20260929 21:02:28.985165 13291 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:02:28.987285 13297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10280e46262c4cd192423aa45512c909. Latest consensus state: current_term: 2 leader_uuid: "10280e46262c4cd192423aa45512c909" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10280e46262c4cd192423aa45512c909" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33225 } } }
I20260929 21:02:28.987702 13297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909 [sys.catalog]: This master's current role is: LEADER
I20260929 21:02:28.988274 13296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "10280e46262c4cd192423aa45512c909" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10280e46262c4cd192423aa45512c909" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33225 } } }
I20260929 21:02:28.988595 13296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909 [sys.catalog]: This master's current role is: LEADER
I20260929 21:02:28.998347 13301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:02:29.005769 13301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:02:29.006646 13291 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 21:02:29.007344 13301 catalog_manager.cc:1259] Loaded cluster ID: 49eea509c10241adb0c0b76e1d1331a5
I20260929 21:02:29.007431 13301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:02:29.012108 13301 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:02:29.014053 13301 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909: Loaded TSK: 0
I20260929 21:02:29.024129 13301 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 21:02:29.360816 13293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:02:29.361316 13293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:02:29.361564 13293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:02:29.373286 13293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:02:29.373485 13293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:02:29.373728 13293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 21:02:29.389266 13293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:50229/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:45259
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=46569
--tserver_master_addrs=127.0.97.62:33225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:02:29.391013 13293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:02:29.392920 13293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:02:29.424902 13328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:29.425599 13329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:29.434939 13331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:02:29.435199 13293 server_base.cc:1034] running on GCE node
I20260929 21:02:29.436157 13293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:02:29.445147 13293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:02:29.448628 13293 hybrid_clock.cc:648] HybridClock initialized: now 1790715749448542 us; error 78 us; skew 500 ppm
I20260929 21:02:29.449110 13293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:02:29 dist-test-slave-1dms krb5kdc[12244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715749, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:02:29.453464 13293 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 21:02:29.460237 13293 webserver.cc:460] Webserver started at http://127.0.97.1:46569/ using document root <none> and password file <none>
I20260929 21:02:29.461280 13293 fs_manager.cc:359] Metadata directory not provided
I20260929 21:02:29.461505 13293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 21:02:29 dist-test-slave-1dms krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715749, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 21:02:29.510939 13293 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.002s sys 0.002s
I20260929 21:02:29.521266 13339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:29.523110 13293 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.005s sys 0.004s
I20260929 21:02:29.523463 13293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-0/wal
uuid: "852489c43e7b45dd90e71259986e0ab2"
format_stamp: "Formatted at 2026-09-29 21:02:26 on dist-test-slave-1dms"
server_key: "b6a0a81ee6835424f0fa0a0a7dce0862"
server_key_iv: "dd0eb1ad816a8997890c00b5f81fa264"
server_key_version: "kuduclusterkey@0"
I20260929 21:02:29.524930 13293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:02:29.574718 13293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:02:29.579854 13293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:02:29.580482 13293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:02:29.587296 13293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:02:29.611022 13293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:02:29.611230 13293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 21:02:29.611390 13293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:02:29.611517 13293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:29.696821 13293 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45259
I20260929 21:02:29.696908 13452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45259 every 8 connection(s)
I20260929 21:02:29.698745 13293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
I20260929 21:02:29.706903 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 13293
I20260929 21:02:29.707815 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:50229/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:36283
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:33225
--webserver_port=35851
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:50229/kms with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 21:02:29 dist-test-slave-1dms krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715749, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:02:29.750414 13456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:29.706804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53655 (local address 127.0.97.62:33225)
0929 21:02:29.723365 (+ 16561us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:29.723387 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:29.724204 (+ 817us) server_negotiation.cc:407] Connection header received
0929 21:02:29.727575 (+ 3371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:29.727600 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:29.728356 (+ 756us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:29.728742 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:29.731277 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:29.733219 (+ 1942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:29.735580 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:29.736580 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:29.736740 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:29.741324 (+ 4584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:29.741334 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:29.741354 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:29.741399 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:29.745155 (+ 3756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:29.746114 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:29.746122 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:29.746129 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:29.746322 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:29.746828 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:29.746832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:29.746835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:29.747439 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:29.747584 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:29.748623 (+ 1039us) server_negotiation.cc:299] Negotiation successful
0929 21:02:29.749071 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16268,"thread_start_us":16057,"threads_started":1}
I20260929 21:02:29.753016 13455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:29.705674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33225 (local address 127.0.97.1:53655)
0929 21:02:29.723993 (+ 18319us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:29.724046 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 21:02:29.726778 (+ 2732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:02:29.729614 (+ 2836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:02:29.729624 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:02:29.730267 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:02:29.731038 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:29.731062 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:29.733434 (+ 2372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:29.733439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:29.735402 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:29.735417 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:29.737226 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:29.737234 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:29.737507 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:29.738327 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:02:29.738358 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:02:29.741014 (+ 2656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:02:29.745349 (+ 4335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:29.745358 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:29.745375 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:29.745931 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:29.746468 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:29.746473 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:29.746476 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:29.746707 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:29.747679 (+ 972us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:02:29.747686 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:02:29.748048 (+ 362us) client_negotiation.cc:769] Sending connection context
0929 21:02:29.751321 (+ 3273us) client_negotiation.cc:240] Negotiation successful
0929 21:02:29.751846 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18012,"thread_start_us":17794,"threads_started":1}
I20260929 21:02:29.756897 13453 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33225
I20260929 21:02:29.757194 13453 heartbeater.cc:464] Registering TS with master...
I20260929 21:02:29.758000 13453 heartbeater.cc:511] Master 127.0.97.62:33225 requested a full tablet report, sending...
W20260929 21:02:29.761273 13453 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:33225 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 21:02:29.761492 13453 heartbeater.cc:464] Registering TS with master...
I20260929 21:02:29.761778 13453 heartbeater.cc:511] Master 127.0.97.62:33225 requested a full tablet report, sending...
I20260929 21:02:29.763140 13453 heartbeater.cc:464] Registering TS with master...
I20260929 21:02:29.763463 13453 heartbeater.cc:511] Master 127.0.97.62:33225 requested a full tablet report, sending...
W20260929 21:02:29.764554 13453 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 21:02:30.027422 13464 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260929 21:02:30.155748 13457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:02:30.156271 13457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:02:30.156458 13457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:02:30.163866 13457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:02:30.164019 13457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:02:30.164225 13457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 21:02:30.175098 13457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:50229/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:36283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=35851
--tserver_master_addrs=127.0.97.62:33225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:02:30.176473 13457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:02:30.178125 13457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:02:30.192551 13469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:30.192750 13470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:30.193495 13472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:02:30.195020 13457 server_base.cc:1034] running on GCE node
I20260929 21:02:30.195684 13457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:02:30.196671 13457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:02:30.197891 13457 hybrid_clock.cc:648] HybridClock initialized: now 1790715750197824 us; error 56 us; skew 500 ppm
I20260929 21:02:30.198196 13457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:02:30 dist-test-slave-1dms krb5kdc[12244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:02:30.201448 13457 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 21:02:30.202996 13457 webserver.cc:460] Webserver started at http://127.0.97.2:35851/ using document root <none> and password file <none>
I20260929 21:02:30.203799 13457 fs_manager.cc:359] Metadata directory not provided
I20260929 21:02:30.204006 13457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 21:02:30 dist-test-slave-1dms krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 21:02:30.231485 13457 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 21:02:30.235064 13482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:30.238314 13457 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260929 21:02:30.238508 13457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-1/wal
uuid: "2a378a26b7254ae6be02700ef74ef773"
format_stamp: "Formatted at 2026-09-29 21:02:27 on dist-test-slave-1dms"
server_key: "342d8782b97334f94ec7aff81d28d24f"
server_key_iv: "b901781b323ea3be2c1d819ccfa4531c"
server_key_version: "kuduclusterkey@0"
I20260929 21:02:30.239784 13457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:02:30.270164 13457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:02:30.275473 13457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:02:30.275832 13457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:02:30.286761 13457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:02:30.294849 13457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:02:30.295011 13457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260929 21:02:30.295157 13457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:02:30.295265 13457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:30.372722 13457 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36283
I20260929 21:02:30.372803 13596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36283 every 8 connection(s)
I20260929 21:02:30.374718 13457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
I20260929 21:02:30.379683 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 13457
I20260929 21:02:30.380448 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:50229/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:41101
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:33225
--webserver_port=45251
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:50229/kms with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 21:02:30 dist-test-slave-1dms krb5kdc[12244](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790715750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 21:02:30 dist-test-slave-1dms krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:02:30.421569 13599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:30.378396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33225 (local address 127.0.97.2:42949)
0929 21:02:30.386750 (+ 8354us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:30.386806 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 21:02:30.389187 (+ 2381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:02:30.390094 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:02:30.390108 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:02:30.390807 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:02:30.391599 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:30.391618 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:30.393855 (+ 2237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:30.393863 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:30.396392 (+ 2529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:30.396410 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:30.397933 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:30.397942 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:30.398297 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:30.403330 (+ 5033us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:02:30.403380 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:02:30.406672 (+ 3292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:02:30.416380 (+ 9708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:30.416393 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:30.416414 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:30.417142 (+ 728us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:30.417652 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:30.417658 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:30.417661 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:30.417923 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:30.418626 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:02:30.418635 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:02:30.419161 (+ 526us) client_negotiation.cc:769] Sending connection context
0929 21:02:30.419502 (+ 341us) client_negotiation.cc:240] Negotiation successful
0929 21:02:30.420133 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":1130,"threads_started":1}
I20260929 21:02:30.422716 13600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:30.379493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42949 (local address 127.0.97.62:33225)
0929 21:02:30.381169 (+ 1676us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:30.381176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:30.389520 (+ 8344us) server_negotiation.cc:407] Connection header received
0929 21:02:30.389595 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:30.389599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:30.389674 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:30.389800 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:30.392297 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:30.393662 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:30.396563 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:30.397405 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:30.397529 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:30.406998 (+ 9469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:30.407006 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:30.407011 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:30.407060 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:30.411033 (+ 3973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:30.417339 (+ 6306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:30.417345 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:30.417350 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:30.417520 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:30.418064 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:30.418069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:30.418073 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:30.418419 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:30.418543 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:30.422102 (+ 3559us) server_negotiation.cc:299] Negotiation successful
0929 21:02:30.422344 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1009,"spinlock_wait_cycles":1051392,"thread_start_us":986,"threads_started":1}
I20260929 21:02:30.424979 13597 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33225
I20260929 21:02:30.425334 13597 heartbeater.cc:464] Registering TS with master...
I20260929 21:02:30.426242 13597 heartbeater.cc:511] Master 127.0.97.62:33225 requested a full tablet report, sending...
I20260929 21:02:30.428591 13256 ts_manager.cc:194] Registered new tserver with Master: 2a378a26b7254ae6be02700ef74ef773 (127.0.97.2:36283)
I20260929 21:02:30.430756 13256 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42949
I20260929 21:02:30.764714 13453 heartbeater.cc:464] Registering TS with master...
I20260929 21:02:30.765213 13453 heartbeater.cc:511] Master 127.0.97.62:33225 requested a full tablet report, sending...
I20260929 21:02:30.766700 13256 ts_manager.cc:194] Registered new tserver with Master: 852489c43e7b45dd90e71259986e0ab2 (127.0.97.1:45259)
I20260929 21:02:30.767890 13256 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53655
W20260929 21:02:30.848853 13601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:02:30.849267 13601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:02:30.849471 13601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:02:30.860849 13601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:02:30.860998 13601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:02:30.861186 13601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 21:02:30.875934 13601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:50229/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:41101
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=45251
--tserver_master_addrs=127.0.97.62:33225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:02:30.877475 13601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:02:30.879004 13601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 29 21:02:30 dist-test-slave-1dms krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260929 21:02:30.917654 13607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:30.928655 13608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:02:30.946447 13601 server_base.cc:1034] running on GCE node
W20260929 21:02:30.951524 13610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:02:30.953161 13601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:02:30.965178 13601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:02:30.966522 13601 hybrid_clock.cc:648] HybridClock initialized: now 1790715750966444 us; error 84 us; skew 500 ppm
I20260929 21:02:30.966976 13601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:02:30 dist-test-slave-1dms krb5kdc[12244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:02:30.971041 13601 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 21:02:30.997262 13601 webserver.cc:460] Webserver started at http://127.0.97.3:45251/ using document root <none> and password file <none>
I20260929 21:02:30.998268 13601 fs_manager.cc:359] Metadata directory not provided
I20260929 21:02:30.998435 13601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 21:02:31 dist-test-slave-1dms krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 21:02:31.019829 13601 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 21:02:31.023499 13618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:31.025512 13601 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 21:02:31.025728 13601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-2/wal
uuid: "73eb589056bd4be3a1a282948eaeb4be"
format_stamp: "Formatted at 2026-09-29 21:02:27 on dist-test-slave-1dms"
server_key: "bef8c1b47e33aab4769cc06db631d49f"
server_key_iv: "e6620b32e10d8530d53bdf487204e3db"
server_key_version: "kuduclusterkey@0"
I20260929 21:02:31.027352 13601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:02:31.078137 13601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:02:31.083774 13601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:02:31.084080 13601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:02:31.085367 13601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:02:31.087231 13601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:02:31.087352 13601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:31.087481 13601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:02:31.087561 13601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:31.135855 13601 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41101
I20260929 21:02:31.135901 13730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41101 every 8 connection(s)
I20260929 21:02:31.138104 13601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
I20260929 21:02:31.138787 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 13601
I20260929 21:02:31.178189 13732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:31.139733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48076 (local address 127.0.97.62:33225)
0929 21:02:31.147220 (+ 7487us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:31.147230 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:31.147259 (+ 29us) server_negotiation.cc:407] Connection header received
0929 21:02:31.147342 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:31.147350 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:31.147452 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:31.147592 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:31.150496 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:31.151969 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:31.163275 (+ 11306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:31.163939 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:31.164082 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:31.168039 (+ 3957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:31.168044 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:31.168047 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:31.168090 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:31.171033 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:31.174932 (+ 3899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:31.174937 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:31.174941 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:31.175107 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:31.175661 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:31.175666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:31.175669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:31.176020 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:31.176299 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:31.177546 (+ 1247us) server_negotiation.cc:299] Negotiation successful
0929 21:02:31.177788 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7284,"thread_start_us":13112,"threads_started":1}
Sep 29 21:02:31 dist-test-slave-1dms krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:02:31.198184 13735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:31.155872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43839 (local address 127.0.97.62:33225)
0929 21:02:31.169079 (+ 13207us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:31.169083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:31.169098 (+ 15us) server_negotiation.cc:407] Connection header received
0929 21:02:31.171225 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:31.171230 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:31.171306 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:31.171399 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:31.173760 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:31.174761 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:31.179738 (+ 4977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:31.180405 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:31.180526 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:31.187025 (+ 6499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:31.187032 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:31.187036 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:31.187075 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:31.190651 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:31.193834 (+ 3183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:31.193838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:31.193841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:31.193963 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:31.194553 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:31.194558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:31.194561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:31.194942 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:31.195085 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:31.197542 (+ 2457us) server_negotiation.cc:299] Negotiation successful
0929 21:02:31.197797 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13052,"thread_start_us":12972,"threads_started":1}
I20260929 21:02:31.199303 13736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:31.156883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33225 (local address 127.0.97.3:43839)
0929 21:02:31.163889 (+ 7006us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:31.163960 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0929 21:02:31.167680 (+ 3720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:02:31.171919 (+ 4239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:02:31.171934 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:02:31.172574 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:02:31.173506 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:31.173537 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:31.174912 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:31.174918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:31.179510 (+ 4592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:31.179523 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:31.180869 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:31.180877 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:31.181185 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:31.184029 (+ 2844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:02:31.184062 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:02:31.186657 (+ 2595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:02:31.190830 (+ 4173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:31.190840 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:31.190859 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:31.193678 (+ 2819us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:31.194078 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:31.194084 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:31.194088 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:31.194400 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:31.195160 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:02:31.195169 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:02:31.195640 (+ 471us) client_negotiation.cc:769] Sending connection context
0929 21:02:31.195974 (+ 334us) client_negotiation.cc:240] Negotiation successful
0929 21:02:31.196579 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6639,"thread_start_us":7207,"threads_started":1}
I20260929 21:02:31.201401 13731 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33225
I20260929 21:02:31.201730 13731 heartbeater.cc:464] Registering TS with master...
I20260929 21:02:31.202577 13731 heartbeater.cc:511] Master 127.0.97.62:33225 requested a full tablet report, sending...
I20260929 21:02:31.204308 13256 ts_manager.cc:194] Registered new tserver with Master: 73eb589056bd4be3a1a282948eaeb4be (127.0.97.3:41101)
I20260929 21:02:31.205834 13256 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:43839
I20260929 21:02:31.207552 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 21:02:31 dist-test-slave-1dms krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 21:02:31.245518 13735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:31.216476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48080 (local address 127.0.97.62:33225)
0929 21:02:31.217099 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:31.217108 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:31.217687 (+ 579us) server_negotiation.cc:407] Connection header received
0929 21:02:31.217891 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:31.217897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:31.217983 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:31.218092 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:31.219803 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:31.220892 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:31.222813 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:31.223487 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:31.223584 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:31.237701 (+ 14117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:31.237710 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:31.237714 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:31.237769 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:31.241351 (+ 3582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:31.242380 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:31.242389 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:31.242395 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:31.242565 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:31.243052 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:31.243057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:31.243060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:31.243454 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:31.243699 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:31.244893 (+ 1194us) server_negotiation.cc:299] Negotiation successful
0929 21:02:31.245119 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438}
I20260929 21:02:31.251775 13256 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 21:02:31.261528 13256 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 21:02:31.391834 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:31.315394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45259 (local address 127.0.0.1:39670)
0929 21:02:31.353630 (+ 38236us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:31.353706 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0929 21:02:31.353943 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:02:31.369187 (+ 15244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:02:31.369195 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:02:31.369233 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:02:31.369662 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:31.369677 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:31.378369 (+ 8692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:31.378375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:31.380622 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:31.380639 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:31.390688 (+ 10049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:31.390695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:31.390945 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:31.391015 (+ 70us) client_negotiation.cc:769] Sending connection context
0929 21:02:31.391184 (+ 169us) client_negotiation.cc:240] Negotiation successful
0929 21:02:31.391419 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37902,"thread_start_us":63840,"threads_started":1}
I20260929 21:02:31.392743 13748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:31.315556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39670 (local address 127.0.97.1:45259)
0929 21:02:31.366988 (+ 51432us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:31.368027 (+ 1039us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:31.368079 (+ 52us) server_negotiation.cc:407] Connection header received
0929 21:02:31.368241 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:31.368254 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:31.368585 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:31.368850 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:02:31.376575 (+ 7725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:31.378187 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:31.388964 (+ 10777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:31.390414 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:31.390555 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:31.390798 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:31.392248 (+ 1450us) server_negotiation.cc:299] Negotiation successful
0929 21:02:31.392432 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51207,"thread_start_us":50406,"threads_started":1}
I20260929 21:02:31.397097 13388 tablet_service.cc:1467] Processing CreateTablet for tablet 5112daf0b343419ea46860c6b6db7a45 (DEFAULT_TABLE table=test-table [id=3657195503734a829bc810cb709b1693]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:02:31.397960 13750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:31.320781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55332 (local address 127.0.97.2:36283)
0929 21:02:31.354697 (+ 33916us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:31.354707 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:31.370862 (+ 16155us) server_negotiation.cc:407] Connection header received
0929 21:02:31.370980 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:31.370988 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:31.371260 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:31.371452 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:02:31.374205 (+ 2753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:31.375873 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:31.382836 (+ 6963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:31.384234 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:31.384371 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:31.384604 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:31.397382 (+ 12778us) server_negotiation.cc:299] Negotiation successful
0929 21:02:31.397595 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33663,"thread_start_us":40952,"threads_started":1}
I20260929 21:02:31.399019 13388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5112daf0b343419ea46860c6b6db7a45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:02:31.400888 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:31.363984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41101 (local address 127.0.0.1:51122)
0929 21:02:31.373306 (+ 9322us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:31.373326 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 21:02:31.373449 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:02:31.384916 (+ 11467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:02:31.384921 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:02:31.384952 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:02:31.385319 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:31.385330 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:31.387580 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:31.387585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:31.391579 (+ 3994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:31.391598 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:31.400055 (+ 8457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:31.400062 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:31.400264 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:31.400299 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 21:02:31.400386 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 21:02:31.400542 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9205,"thread_start_us":28124,"threads_started":1}
I20260929 21:02:31.401316 13747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:31.320262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36283 (local address 127.0.0.1:55332)
0929 21:02:31.370593 (+ 50331us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:31.370621 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 21:02:31.370743 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:02:31.373571 (+ 2828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:02:31.373575 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:02:31.373601 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:02:31.373954 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:31.373968 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:31.380797 (+ 6829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:31.380802 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:31.382649 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:31.382662 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:31.391871 (+ 9209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:31.391878 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:31.392439 (+ 561us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:31.392478 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 21:02:31.392552 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 21:02:31.392685 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50193,"thread_start_us":34454,"threads_started":1}
I20260929 21:02:31.401703 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:31.356018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51122 (local address 127.0.97.3:41101)
0929 21:02:31.372711 (+ 16693us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:31.372720 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:31.376044 (+ 3324us) server_negotiation.cc:407] Connection header received
0929 21:02:31.376152 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:31.376160 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:31.376395 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:31.385465 (+ 9070us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:02:31.385808 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:31.387420 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:31.398322 (+ 10902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:31.399797 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:31.399975 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:31.400201 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:31.401147 (+ 946us) server_negotiation.cc:299] Negotiation successful
0929 21:02:31.401346 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16497,"thread_start_us":16827,"threads_started":1}
I20260929 21:02:31.405640 13532 tablet_service.cc:1467] Processing CreateTablet for tablet 5112daf0b343419ea46860c6b6db7a45 (DEFAULT_TABLE table=test-table [id=3657195503734a829bc810cb709b1693]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:02:31.405712 13666 tablet_service.cc:1467] Processing CreateTablet for tablet 5112daf0b343419ea46860c6b6db7a45 (DEFAULT_TABLE table=test-table [id=3657195503734a829bc810cb709b1693]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:02:31.407176 13666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5112daf0b343419ea46860c6b6db7a45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:02:31.407208 13532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5112daf0b343419ea46860c6b6db7a45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:02:31.435845 13597 heartbeater.cc:503] Master 127.0.97.62:33225 was elected leader, sending a full tablet report...
I20260929 21:02:31.443742 13753 tablet_bootstrap.cc:492] T 5112daf0b343419ea46860c6b6db7a45 P 2a378a26b7254ae6be02700ef74ef773: Bootstrap starting.
I20260929 21:02:31.449823 13755 tablet_bootstrap.cc:492] T 5112daf0b343419ea46860c6b6db7a45 P 852489c43e7b45dd90e71259986e0ab2: Bootstrap starting.
I20260929 21:02:31.451985 13754 tablet_bootstrap.cc:492] T 5112daf0b343419ea46860c6b6db7a45 P 73eb589056bd4be3a1a282948eaeb4be: Bootstrap starting.
I20260929 21:02:31.455629 13755 tablet_bootstrap.cc:654] T 5112daf0b343419ea46860c6b6db7a45 P 852489c43e7b45dd90e71259986e0ab2: Neither blocks nor log segments found. Creating new log.
I20260929 21:02:31.459683 13753 tablet_bootstrap.cc:654] T 5112daf0b343419ea46860c6b6db7a45 P 2a378a26b7254ae6be02700ef74ef773: Neither blocks nor log segments found. Creating new log.
I20260929 21:02:31.461064 13753 log.cc:824] T 5112daf0b343419ea46860c6b6db7a45 P 2a378a26b7254ae6be02700ef74ef773: Log is configured to *not* fsync() on all Append() calls
I20260929 21:02:31.463069 13754 tablet_bootstrap.cc:654] T 5112daf0b343419ea46860c6b6db7a45 P 73eb589056bd4be3a1a282948eaeb4be: Neither blocks nor log segments found. Creating new log.
I20260929 21:02:31.464336 13754 log.cc:824] T 5112daf0b343419ea46860c6b6db7a45 P 73eb589056bd4be3a1a282948eaeb4be: Log is configured to *not* fsync() on all Append() calls
I20260929 21:02:31.465976 13755 log.cc:824] T 5112daf0b343419ea46860c6b6db7a45 P 852489c43e7b45dd90e71259986e0ab2: Log is configured to *not* fsync() on all Append() calls
I20260929 21:02:31.470826 13753 tablet_bootstrap.cc:492] T 5112daf0b343419ea46860c6b6db7a45 P 2a378a26b7254ae6be02700ef74ef773: No bootstrap required, opened a new log
I20260929 21:02:31.471168 13753 ts_tablet_manager.cc:1397] T 5112daf0b343419ea46860c6b6db7a45 P 2a378a26b7254ae6be02700ef74ef773: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.004s
I20260929 21:02:31.481050 13755 tablet_bootstrap.cc:492] T 5112daf0b343419ea46860c6b6db7a45 P 852489c43e7b45dd90e71259986e0ab2: No bootstrap required, opened a new log
I20260929 21:02:31.482369 13755 ts_tablet_manager.cc:1397] T 5112daf0b343419ea46860c6b6db7a45 P 852489c43e7b45dd90e71259986e0ab2: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.009s
I20260929 21:02:31.483855 13754 tablet_bootstrap.cc:492] T 5112daf0b343419ea46860c6b6db7a45 P 73eb589056bd4be3a1a282948eaeb4be: No bootstrap required, opened a new log
I20260929 21:02:31.484197 13754 ts_tablet_manager.cc:1397] T 5112daf0b343419ea46860c6b6db7a45 P 73eb589056bd4be3a1a282948eaeb4be: Time spent bootstrapping tablet: real 0.032s user 0.011s sys 0.006s
I20260929 21:02:31.493456 13755 raft_consensus.cc:348] T 5112daf0b343419ea46860c6b6db7a45 P 852489c43e7b45dd90e71259986e0ab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852489c43e7b45dd90e71259986e0ab2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45259 } } peers { permanent_uuid: "2a378a26b7254ae6be02700ef74ef773" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36283 } } peers { permanent_uuid: "73eb589056bd4be3a1a282948eaeb4be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41101 } }
I20260929 21:02:31.494036 13755 raft_consensus.cc:374] T 5112daf0b343419ea46860c6b6db7a45 P 852489c43e7b45dd90e71259986e0ab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:02:31.494194 13755 raft_consensus.cc:729] T 5112daf0b343419ea46860c6b6db7a45 P 852489c43e7b45dd90e71259986e0ab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 852489c43e7b45dd90e71259986e0ab2, State: Initialized, Role: FOLLOWER
I20260929 21:02:31.494975 13755 consensus_queue.cc:260] T 5112daf0b343419ea46860c6b6db7a45 P 852489c43e7b45dd90e71259986e0ab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852489c43e7b45dd90e71259986e0ab2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45259 } } peers { permanent_uuid: "2a378a26b7254ae6be02700ef74ef773" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36283 } } peers { permanent_uuid: "73eb589056bd4be3a1a282948eaeb4be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41101 } }
I20260929 21:02:31.499989 13753 raft_consensus.cc:348] T 5112daf0b343419ea46860c6b6db7a45 P 2a378a26b7254ae6be02700ef74ef773 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852489c43e7b45dd90e71259986e0ab2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45259 } } peers { permanent_uuid: "2a378a26b7254ae6be02700ef74ef773" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36283 } } peers { permanent_uuid: "73eb589056bd4be3a1a282948eaeb4be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41101 } }
I20260929 21:02:31.500486 13753 raft_consensus.cc:374] T 5112daf0b343419ea46860c6b6db7a45 P 2a378a26b7254ae6be02700ef74ef773 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:02:31.500629 13753 raft_consensus.cc:729] T 5112daf0b343419ea46860c6b6db7a45 P 2a378a26b7254ae6be02700ef74ef773 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a378a26b7254ae6be02700ef74ef773, State: Initialized, Role: FOLLOWER
I20260929 21:02:31.501366 13753 consensus_queue.cc:260] T 5112daf0b343419ea46860c6b6db7a45 P 2a378a26b7254ae6be02700ef74ef773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852489c43e7b45dd90e71259986e0ab2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45259 } } peers { permanent_uuid: "2a378a26b7254ae6be02700ef74ef773" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36283 } } peers { permanent_uuid: "73eb589056bd4be3a1a282948eaeb4be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41101 } }
I20260929 21:02:31.494585 13754 raft_consensus.cc:348] T 5112daf0b343419ea46860c6b6db7a45 P 73eb589056bd4be3a1a282948eaeb4be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852489c43e7b45dd90e71259986e0ab2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45259 } } peers { permanent_uuid: "2a378a26b7254ae6be02700ef74ef773" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36283 } } peers { permanent_uuid: "73eb589056bd4be3a1a282948eaeb4be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41101 } }
I20260929 21:02:31.502019 13754 raft_consensus.cc:374] T 5112daf0b343419ea46860c6b6db7a45 P 73eb589056bd4be3a1a282948eaeb4be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:02:31.502231 13754 raft_consensus.cc:729] T 5112daf0b343419ea46860c6b6db7a45 P 73eb589056bd4be3a1a282948eaeb4be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73eb589056bd4be3a1a282948eaeb4be, State: Initialized, Role: FOLLOWER
I20260929 21:02:31.502990 13754 consensus_queue.cc:260] T 5112daf0b343419ea46860c6b6db7a45 P 73eb589056bd4be3a1a282948eaeb4be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852489c43e7b45dd90e71259986e0ab2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45259 } } peers { permanent_uuid: "2a378a26b7254ae6be02700ef74ef773" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36283 } } peers { permanent_uuid: "73eb589056bd4be3a1a282948eaeb4be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41101 } }
I20260929 21:02:31.505103 13755 ts_tablet_manager.cc:1428] T 5112daf0b343419ea46860c6b6db7a45 P 852489c43e7b45dd90e71259986e0ab2: Time spent starting tablet: real 0.023s user 0.017s sys 0.002s
I20260929 21:02:31.505499 13453 heartbeater.cc:503] Master 127.0.97.62:33225 was elected leader, sending a full tablet report...
I20260929 21:02:31.512421 13753 ts_tablet_manager.cc:1428] T 5112daf0b343419ea46860c6b6db7a45 P 2a378a26b7254ae6be02700ef74ef773: Time spent starting tablet: real 0.041s user 0.008s sys 0.009s
I20260929 21:02:31.516803 13731 heartbeater.cc:503] Master 127.0.97.62:33225 was elected leader, sending a full tablet report...
I20260929 21:02:31.517553 13754 ts_tablet_manager.cc:1428] T 5112daf0b343419ea46860c6b6db7a45 P 73eb589056bd4be3a1a282948eaeb4be: Time spent starting tablet: real 0.033s user 0.010s sys 0.010s
W20260929 21:02:31.637233 13598 tablet.cc:2369] T 5112daf0b343419ea46860c6b6db7a45 P 2a378a26b7254ae6be02700ef74ef773: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 21:02:31.660987 13734 tablet.cc:2369] T 5112daf0b343419ea46860c6b6db7a45 P 73eb589056bd4be3a1a282948eaeb4be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 21:02:31.699836 13761 raft_consensus.cc:482] T 5112daf0b343419ea46860c6b6db7a45 P 73eb589056bd4be3a1a282948eaeb4be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 21:02:31.700201 13761 raft_consensus.cc:504] T 5112daf0b343419ea46860c6b6db7a45 P 73eb589056bd4be3a1a282948eaeb4be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852489c43e7b45dd90e71259986e0ab2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45259 } } peers { permanent_uuid: "2a378a26b7254ae6be02700ef74ef773" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36283 } } peers { permanent_uuid: "73eb589056bd4be3a1a282948eaeb4be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41101 } }
I20260929 21:02:31.702236 13761 leader_election.cc:288] T 5112daf0b343419ea46860c6b6db7a45 P 73eb589056bd4be3a1a282948eaeb4be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 852489c43e7b45dd90e71259986e0ab2 (127.0.97.1:45259), 2a378a26b7254ae6be02700ef74ef773 (127.0.97.2:36283)
W20260929 21:02:31.706359 13454 tablet.cc:2369] T 5112daf0b343419ea46860c6b6db7a45 P 852489c43e7b45dd90e71259986e0ab2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 21:02:31.717712 13750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:31.703109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50847 (local address 127.0.97.2:36283)
0929 21:02:31.703311 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:31.703317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:31.711464 (+ 8147us) server_negotiation.cc:407] Connection header received
0929 21:02:31.711739 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:31.711747 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:31.711825 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:31.711889 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:02:31.712642 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:31.713561 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:31.715443 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:31.716697 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:31.716874 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:31.716981 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:31.717315 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 21:02:31.717460 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 21:02:31.717770 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:31.702653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36283 (local address 127.0.97.3:50847)
0929 21:02:31.711335 (+ 8682us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:31.711359 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 21:02:31.711565 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:02:31.712105 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:02:31.712110 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:02:31.712135 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:02:31.712466 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:31.712477 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:31.713702 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:31.713712 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:31.715287 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:31.715303 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:31.716908 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:31.716914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:31.717124 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:31.717164 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 21:02:31.717274 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 21:02:31.717406 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8561,"mutex_wait_us":47,"thread_start_us":8064,"threads_started":1}
I20260929 21:02:31.717890 13748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:31.702874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52947 (local address 127.0.97.1:45259)
0929 21:02:31.703067 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:31.703076 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:31.711200 (+ 8124us) server_negotiation.cc:407] Connection header received
0929 21:02:31.711508 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:31.711512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:31.711572 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:31.711676 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:02:31.712156 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:31.713113 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:31.715562 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:31.716746 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:31.716898 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:31.716986 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:31.717510 (+ 524us) server_negotiation.cc:299] Negotiation successful
0929 21:02:31.717639 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 21:02:31.718639 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:31.702442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45259 (local address 127.0.97.3:52947)
0929 21:02:31.711054 (+ 8612us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:31.711082 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 21:02:31.711325 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:02:31.711704 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:02:31.711710 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:02:31.711736 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:02:31.711986 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:31.711995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:31.713266 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:31.713271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:31.715394 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:31.715408 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:31.717144 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:31.717149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:31.717325 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:31.717356 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 21:02:31.718104 (+ 748us) client_negotiation.cc:240] Negotiation successful
0929 21:02:31.718251 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8418,"mutex_wait_us":74,"thread_start_us":8634,"threads_started":1}
I20260929 21:02:31.719142 13552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5112daf0b343419ea46860c6b6db7a45" candidate_uuid: "73eb589056bd4be3a1a282948eaeb4be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a378a26b7254ae6be02700ef74ef773" is_pre_election: true
I20260929 21:02:31.719709 13552 raft_consensus.cc:2454] T 5112daf0b343419ea46860c6b6db7a45 P 2a378a26b7254ae6be02700ef74ef773 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73eb589056bd4be3a1a282948eaeb4be in term 0.
I20260929 21:02:31.719884 13408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5112daf0b343419ea46860c6b6db7a45" candidate_uuid: "73eb589056bd4be3a1a282948eaeb4be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "852489c43e7b45dd90e71259986e0ab2" is_pre_election: true
I20260929 21:02:31.720441 13619 leader_election.cc:302] T 5112daf0b343419ea46860c6b6db7a45 P 73eb589056bd4be3a1a282948eaeb4be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a378a26b7254ae6be02700ef74ef773, 73eb589056bd4be3a1a282948eaeb4be; no voters:
I20260929 21:02:31.720438 13408 raft_consensus.cc:2454] T 5112daf0b343419ea46860c6b6db7a45 P 852489c43e7b45dd90e71259986e0ab2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73eb589056bd4be3a1a282948eaeb4be in term 0.
I20260929 21:02:31.720795 13761 raft_consensus.cc:2781] T 5112daf0b343419ea46860c6b6db7a45 P 73eb589056bd4be3a1a282948eaeb4be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 21:02:31.720981 13761 raft_consensus.cc:482] T 5112daf0b343419ea46860c6b6db7a45 P 73eb589056bd4be3a1a282948eaeb4be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 21:02:31.721139 13761 raft_consensus.cc:3037] T 5112daf0b343419ea46860c6b6db7a45 P 73eb589056bd4be3a1a282948eaeb4be [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:02:31.723300 13761 raft_consensus.cc:504] T 5112daf0b343419ea46860c6b6db7a45 P 73eb589056bd4be3a1a282948eaeb4be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852489c43e7b45dd90e71259986e0ab2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45259 } } peers { permanent_uuid: "2a378a26b7254ae6be02700ef74ef773" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36283 } } peers { permanent_uuid: "73eb589056bd4be3a1a282948eaeb4be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41101 } }
I20260929 21:02:31.723904 13761 leader_election.cc:288] T 5112daf0b343419ea46860c6b6db7a45 P 73eb589056bd4be3a1a282948eaeb4be [CANDIDATE]: Term 1 election: Requested vote from peers 852489c43e7b45dd90e71259986e0ab2 (127.0.97.1:45259), 2a378a26b7254ae6be02700ef74ef773 (127.0.97.2:36283)
I20260929 21:02:31.724298 13408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5112daf0b343419ea46860c6b6db7a45" candidate_uuid: "73eb589056bd4be3a1a282948eaeb4be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "852489c43e7b45dd90e71259986e0ab2"
I20260929 21:02:31.724421 13552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5112daf0b343419ea46860c6b6db7a45" candidate_uuid: "73eb589056bd4be3a1a282948eaeb4be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a378a26b7254ae6be02700ef74ef773"
I20260929 21:02:31.724486 13408 raft_consensus.cc:3037] T 5112daf0b343419ea46860c6b6db7a45 P 852489c43e7b45dd90e71259986e0ab2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:02:31.724627 13552 raft_consensus.cc:3037] T 5112daf0b343419ea46860c6b6db7a45 P 2a378a26b7254ae6be02700ef74ef773 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:02:31.727241 13408 raft_consensus.cc:2454] T 5112daf0b343419ea46860c6b6db7a45 P 852489c43e7b45dd90e71259986e0ab2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73eb589056bd4be3a1a282948eaeb4be in term 1.
I20260929 21:02:31.727638 13620 leader_election.cc:302] T 5112daf0b343419ea46860c6b6db7a45 P 73eb589056bd4be3a1a282948eaeb4be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73eb589056bd4be3a1a282948eaeb4be, 852489c43e7b45dd90e71259986e0ab2; no voters:
I20260929 21:02:31.727880 13552 raft_consensus.cc:2454] T 5112daf0b343419ea46860c6b6db7a45 P 2a378a26b7254ae6be02700ef74ef773 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73eb589056bd4be3a1a282948eaeb4be in term 1.
I20260929 21:02:31.727955 13761 raft_consensus.cc:2781] T 5112daf0b343419ea46860c6b6db7a45 P 73eb589056bd4be3a1a282948eaeb4be [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:02:31.728525 13761 raft_consensus.cc:686] T 5112daf0b343419ea46860c6b6db7a45 P 73eb589056bd4be3a1a282948eaeb4be [term 1 LEADER]: Becoming Leader. State: Replica: 73eb589056bd4be3a1a282948eaeb4be, State: Running, Role: LEADER
I20260929 21:02:31.729063 13761 consensus_queue.cc:237] T 5112daf0b343419ea46860c6b6db7a45 P 73eb589056bd4be3a1a282948eaeb4be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852489c43e7b45dd90e71259986e0ab2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45259 } } peers { permanent_uuid: "2a378a26b7254ae6be02700ef74ef773" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36283 } } peers { permanent_uuid: "73eb589056bd4be3a1a282948eaeb4be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41101 } }
I20260929 21:02:31.734494 13256 catalog_manager.cc:5495] T 5112daf0b343419ea46860c6b6db7a45 P 73eb589056bd4be3a1a282948eaeb4be reported cstate change: term changed from 0 to 1, leader changed from <none> to 73eb589056bd4be3a1a282948eaeb4be (127.0.97.3). New cstate: current_term: 1 leader_uuid: "73eb589056bd4be3a1a282948eaeb4be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852489c43e7b45dd90e71259986e0ab2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a378a26b7254ae6be02700ef74ef773" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73eb589056bd4be3a1a282948eaeb4be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41101 } health_report { overall_health: HEALTHY } } }
I20260929 21:02:31.758945 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:31.752539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51136 (local address 127.0.97.3:41101)
0929 21:02:31.752766 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:31.752772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:31.753108 (+ 336us) server_negotiation.cc:407] Connection header received
0929 21:02:31.753309 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:31.753315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:31.753412 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:31.753574 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:02:31.753936 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:31.755050 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:31.756400 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:31.757044 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:31.757180 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:31.757653 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:02:31.758361 (+ 708us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:02:31.758517 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:31.758596 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 21:02:31.758679 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 21:02:31.768481 13552 raft_consensus.cc:1261] T 5112daf0b343419ea46860c6b6db7a45 P 2a378a26b7254ae6be02700ef74ef773 [term 1 FOLLOWER]: Refusing update from remote peer 73eb589056bd4be3a1a282948eaeb4be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:02:31.768616 13408 raft_consensus.cc:1261] T 5112daf0b343419ea46860c6b6db7a45 P 852489c43e7b45dd90e71259986e0ab2 [term 1 FOLLOWER]: Refusing update from remote peer 73eb589056bd4be3a1a282948eaeb4be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:02:31.769613 13771 consensus_queue.cc:1035] T 5112daf0b343419ea46860c6b6db7a45 P 73eb589056bd4be3a1a282948eaeb4be [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a378a26b7254ae6be02700ef74ef773" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:02:31.769976 13761 consensus_queue.cc:1035] T 5112daf0b343419ea46860c6b6db7a45 P 73eb589056bd4be3a1a282948eaeb4be [LEADER]: Connected to new peer: Peer: permanent_uuid: "852489c43e7b45dd90e71259986e0ab2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:02:31.784631 13775 mvcc.cc:205] Tried to move back new op lower bound from 7334771719232630784 to 7334771719087341568. Current Snapshot: MvccSnapshot[applied={T|T < 7334771719232630784}]
I20260929 21:02:31.792434 13777 mvcc.cc:205] Tried to move back new op lower bound from 7334771719232630784 to 7334771719087341568. Current Snapshot: MvccSnapshot[applied={T|T < 7334771719232630784}]
I20260929 21:02:31.810674 13255 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 21:02:31.810957 13255 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 21:02:31.814885 13255 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 10280e46262c4cd192423aa45512c909: Sending DeleteTablet for 3 replicas of tablet 5112daf0b343419ea46860c6b6db7a45
I20260929 21:02:31.816032 13388 tablet_service.cc:1514] Processing DeleteTablet for tablet 5112daf0b343419ea46860c6b6db7a45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:02:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39670
I20260929 21:02:31.816965 13784 tablet_replica.cc:330] T 5112daf0b343419ea46860c6b6db7a45 P 852489c43e7b45dd90e71259986e0ab2: stopping tablet replica
I20260929 21:02:31.817528 13532 tablet_service.cc:1514] Processing DeleteTablet for tablet 5112daf0b343419ea46860c6b6db7a45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:02:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55332
I20260929 21:02:31.817827 13784 raft_consensus.cc:2229] T 5112daf0b343419ea46860c6b6db7a45 P 852489c43e7b45dd90e71259986e0ab2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:02:31.817942 13666 tablet_service.cc:1514] Processing DeleteTablet for tablet 5112daf0b343419ea46860c6b6db7a45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:02:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51122
I20260929 21:02:31.818392 13784 raft_consensus.cc:2258] T 5112daf0b343419ea46860c6b6db7a45 P 852489c43e7b45dd90e71259986e0ab2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:02:31.821727 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 13293
I20260929 21:02:31.833532 13785 tablet_replica.cc:330] T 5112daf0b343419ea46860c6b6db7a45 P 2a378a26b7254ae6be02700ef74ef773: stopping tablet replica
I20260929 21:02:31.833706 13786 tablet_replica.cc:330] T 5112daf0b343419ea46860c6b6db7a45 P 73eb589056bd4be3a1a282948eaeb4be: stopping tablet replica
I20260929 21:02:31.834022 13785 raft_consensus.cc:2229] T 5112daf0b343419ea46860c6b6db7a45 P 2a378a26b7254ae6be02700ef74ef773 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:02:31.834161 13786 raft_consensus.cc:2229] T 5112daf0b343419ea46860c6b6db7a45 P 73eb589056bd4be3a1a282948eaeb4be [term 1 LEADER]: Raft consensus shutting down.
I20260929 21:02:31.834487 13784 ts_tablet_manager.cc:1905] T 5112daf0b343419ea46860c6b6db7a45 P 852489c43e7b45dd90e71259986e0ab2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:02:31.834690 13785 raft_consensus.cc:2258] T 5112daf0b343419ea46860c6b6db7a45 P 2a378a26b7254ae6be02700ef74ef773 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:02:31.834956 13786 raft_consensus.cc:2258] T 5112daf0b343419ea46860c6b6db7a45 P 73eb589056bd4be3a1a282948eaeb4be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:02:31.836813 13785 ts_tablet_manager.cc:1905] T 5112daf0b343419ea46860c6b6db7a45 P 2a378a26b7254ae6be02700ef74ef773: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:02:31.837102 13786 ts_tablet_manager.cc:1905] T 5112daf0b343419ea46860c6b6db7a45 P 73eb589056bd4be3a1a282948eaeb4be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:02:31.844040 13785 ts_tablet_manager.cc:1918] T 5112daf0b343419ea46860c6b6db7a45 P 2a378a26b7254ae6be02700ef74ef773: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:02:31.844444 13786 ts_tablet_manager.cc:1918] T 5112daf0b343419ea46860c6b6db7a45 P 73eb589056bd4be3a1a282948eaeb4be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:02:31.844728 13786 log.cc:1196] T 5112daf0b343419ea46860c6b6db7a45 P 73eb589056bd4be3a1a282948eaeb4be: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-2/wal/wals/5112daf0b343419ea46860c6b6db7a45
I20260929 21:02:31.845497 13786 ts_tablet_manager.cc:1939] T 5112daf0b343419ea46860c6b6db7a45 P 73eb589056bd4be3a1a282948eaeb4be: Deleting consensus metadata
I20260929 21:02:31.845588 13785 log.cc:1196] T 5112daf0b343419ea46860c6b6db7a45 P 2a378a26b7254ae6be02700ef74ef773: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-1/wal/wals/5112daf0b343419ea46860c6b6db7a45
I20260929 21:02:31.846159 13785 ts_tablet_manager.cc:1939] T 5112daf0b343419ea46860c6b6db7a45 P 2a378a26b7254ae6be02700ef74ef773: Deleting consensus metadata
I20260929 21:02:31.847908 13240 catalog_manager.cc:4841] TS 73eb589056bd4be3a1a282948eaeb4be (127.0.97.3:41101): tablet 5112daf0b343419ea46860c6b6db7a45 (table test-table [id=3657195503734a829bc810cb709b1693]) successfully deleted
I20260929 21:02:31.848253 13240 catalog_manager.cc:4841] TS 2a378a26b7254ae6be02700ef74ef773 (127.0.97.2:36283): tablet 5112daf0b343419ea46860c6b6db7a45 (table test-table [id=3657195503734a829bc810cb709b1693]) successfully deleted
I20260929 21:02:31.853088 13748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:31.823042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39680 (local address 127.0.97.1:45259)
0929 21:02:31.823227 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:31.823233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:31.829324 (+ 6091us) server_negotiation.cc:407] Connection header received
0929 21:02:31.829612 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:31.829620 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:31.829729 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:31.829823 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:31.832017 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:31.833455 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:31.836957 (+ 3502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:31.837618 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:31.837718 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:31.845041 (+ 7323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:31.845059 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:31.845078 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:31.845134 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:31.849333 (+ 4199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:31.850258 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:31.850270 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:31.850276 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:31.850453 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:31.850941 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:31.850945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:31.850948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:31.851415 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:31.851751 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:31.852367 (+ 616us) server_negotiation.cc:299] Negotiation successful
0929 21:02:31.852633 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 21:02:31.853914 13784 ts_tablet_manager.cc:1918] T 5112daf0b343419ea46860c6b6db7a45 P 852489c43e7b45dd90e71259986e0ab2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:02:31.854249 13784 log.cc:1196] T 5112daf0b343419ea46860c6b6db7a45 P 852489c43e7b45dd90e71259986e0ab2: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790715556572573-388-0/minicluster-data/ts-0/wal/wals/5112daf0b343419ea46860c6b6db7a45
I20260929 21:02:31.854908 13784 ts_tablet_manager.cc:1939] T 5112daf0b343419ea46860c6b6db7a45 P 852489c43e7b45dd90e71259986e0ab2: Deleting consensus metadata
I20260929 21:02:31.854934 13448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:02:31.964716 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 13293
I20260929 21:02:31.983541 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 13457
W20260929 21:02:31.983441 13241 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:45259 (error 108)
I20260929 21:02:32.007309 13750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:31.984361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55338 (local address 127.0.97.2:36283)
0929 21:02:31.984550 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:31.984556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:31.984663 (+ 107us) server_negotiation.cc:407] Connection header received
0929 21:02:31.984910 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:31.984921 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:31.985033 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:31.985139 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:31.986651 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:31.988098 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:31.991004 (+ 2906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:31.991900 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:31.992046 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:31.996599 (+ 4553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:31.996667 (+ 68us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:31.996702 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:31.996768 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:32.002036 (+ 5268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:32.003211 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:32.003225 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:32.003231 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:32.003444 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:32.003987 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:32.003995 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:32.004000 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:32.004534 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:32.006158 (+ 1624us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:32.006363 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 21:02:32.006673 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 21:02:32.008538 13592 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:02:32.022372 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:32.001954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45259 (local address 127.0.0.1:39688)
0929 21:02:32.021879 (+ 19925us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:32.022075 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19780,"thread_start_us":19647,"threads_started":1}
W20260929 21:02:32.023522 13241 catalog_manager.cc:4584] TS 852489c43e7b45dd90e71259986e0ab2 (127.0.97.1:45259): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5112daf0b343419ea46860c6b6db7a45: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45259: connect: Connection refused (error 111)
I20260929 21:02:32.077632 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:32.077055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45259 (local address 127.0.0.1:39698)
0929 21:02:32.077354 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:32.077457 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260929 21:02:32.078922 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:32.078532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45259 (local address 127.0.0.1:39708)
0929 21:02:32.078709 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:32.078781 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 21:02:32.111692 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 13457
I20260929 21:02:32.128868 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 13601
I20260929 21:02:32.139307 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:32.138786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45259 (local address 127.0.0.1:39714)
0929 21:02:32.139038 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:32.139139 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260929 21:02:32.140506 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:32.140106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45259 (local address 127.0.0.1:39730)
0929 21:02:32.140295 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:32.140366 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260929 21:02:32.149853 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:32.130102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51152 (local address 127.0.97.3:41101)
0929 21:02:32.130276 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:32.130280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:32.130296 (+ 16us) server_negotiation.cc:407] Connection header received
0929 21:02:32.130349 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:32.130352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:32.130411 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:32.130537 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:32.131956 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:32.133013 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:32.134719 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:32.135411 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:32.135542 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:32.141717 (+ 6175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:32.141729 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:32.141745 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:32.141788 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:32.145966 (+ 4178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:32.146795 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:32.146804 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:32.146806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:32.146972 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:32.147414 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:32.147425 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:32.147429 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:32.147862 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:32.148138 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:32.149291 (+ 1153us) server_negotiation.cc:299] Negotiation successful
0929 21:02:32.149492 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 21:02:32.150831 13726 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:02:32.221359 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:32.220813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45259 (local address 127.0.0.1:39742)
0929 21:02:32.221069 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:32.221174 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260929 21:02:32.222528 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:32.222080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45259 (local address 127.0.0.1:39748)
0929 21:02:32.222287 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:32.222385 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260929 21:02:32.270103 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 13601
I20260929 21:02:32.289450 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 13223
I20260929 21:02:32.290612 13286 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:02:32.404448 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 13223
I20260929 21:02:32.404533 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:32.403799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45259 (local address 127.0.0.1:39762)
0929 21:02:32.404226 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:32.404328 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238}
I20260929 21:02:32.426417 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 21:02:32.771347 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 21:02:32.771585 388 mini_ranger.cc:66] Stopping Ranger...
I20260929 21:02:33.116285 388 mini_ranger.cc:68] Stopped Ranger
2026-09-29 21:02:33.116 UTC [12274] LOG: received smart shutdown request
2026-09-29 21:02:33.119 UTC [12274] LOG: background worker "logical replication launcher" (PID 12283) exited with exit code 1
2026-09-29 21:02:33.120 UTC [12278] LOG: shutting down
2026-09-29 21:02:33.296 UTC [12274] LOG: database system is shut down
2026-09-29T21:02:33Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52773 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 21:02:33 dist-test-slave-1dms krb5kdc[13802](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 21:02:33 dist-test-slave-1dms krb5kdc[13802](info): set up 2 sockets
Sep 29 21:02:33 dist-test-slave-1dms krb5kdc[13802](info): commencing operation
krb5kdc: starting...
W20260929 21:02:34.537364 388 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 21:02:34 dist-test-slave-1dms krb5kdc[13802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T21:02:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:02:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 21:02:34.756600 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38435
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38435,127.0.97.61:36597,127.0.97.60:39675
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:02:35.076978 13815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:02:35.077270 13815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:02:35.077359 13815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:02:35.084410 13815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:02:35.084517 13815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:02:35.084563 13815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:02:35.084619 13815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:02:35.084662 13815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:02:35.094347 13815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38435,127.0.97.61:36597,127.0.97.60:39675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:02:35.095641 13815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:02:35.096946 13815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:02:35.108611 13821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:35.108669 13823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:35.108572 13820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:02:35.109815 13815 server_base.cc:1034] running on GCE node
I20260929 21:02:35.110571 13815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:02:35.111927 13815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:02:35.113153 13815 hybrid_clock.cc:648] HybridClock initialized: now 1790715755113116 us; error 52 us; skew 500 ppm
I20260929 21:02:35.113499 13815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:02:35 dist-test-slave-1dms krb5kdc[13802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:02:35.123461 13815 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 21:02:35.125097 13815 webserver.cc:460] Webserver started at http://127.0.97.62:43853/ using document root <none> and password file <none>
I20260929 21:02:35.125824 13815 fs_manager.cc:359] Metadata directory not provided
I20260929 21:02:35.125972 13815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:02:35.126302 13815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:02:35.129379 13815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "6eb9d6542efb47588bc0246f1e9c7c4c"
format_stamp: "Formatted at 2026-09-29 21:02:35 on dist-test-slave-1dms"
server_key: "0b9690dea6986191c19bf191b58bbdba"
server_key_iv: "2c17a6e4493ae2344576c8141e22e495"
server_key_version: "encryptionkey@0"
I20260929 21:02:35.130183 13815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "6eb9d6542efb47588bc0246f1e9c7c4c"
format_stamp: "Formatted at 2026-09-29 21:02:35 on dist-test-slave-1dms"
server_key: "0b9690dea6986191c19bf191b58bbdba"
server_key_iv: "2c17a6e4493ae2344576c8141e22e495"
server_key_version: "encryptionkey@0"
I20260929 21:02:35.135357 13815 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 21:02:35.138934 13830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:35.140306 13815 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 21:02:35.140484 13815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "6eb9d6542efb47588bc0246f1e9c7c4c"
format_stamp: "Formatted at 2026-09-29 21:02:35 on dist-test-slave-1dms"
server_key: "0b9690dea6986191c19bf191b58bbdba"
server_key_iv: "2c17a6e4493ae2344576c8141e22e495"
server_key_version: "encryptionkey@0"
I20260929 21:02:35.140683 13815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:02:35.166236 13815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:02:35.170334 13815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:02:35.170651 13815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:02:35.188906 13815 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38435
I20260929 21:02:35.188930 13881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38435 every 8 connection(s)
I20260929 21:02:35.190294 13815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:02:35.193959 13882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:02:35.197280 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 13815
I20260929 21:02:35.197530 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:02:35.198233 388 external_mini_cluster.cc:1428] Setting key 21bcbaf48cb24bbbebb1dbbb9fa19790
I20260929 21:02:35.200443 13882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38435 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 21:02:35 dist-test-slave-1dms krb5kdc[13802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260929 21:02:35.223805 13884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:35.203659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38110 (local address 127.0.97.62:38435)
0929 21:02:35.204548 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:35.204568 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:35.205028 (+ 460us) server_negotiation.cc:407] Connection header received
0929 21:02:35.206975 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:35.206981 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:35.207196 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:35.207400 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:35.208634 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:35.209865 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:35.211205 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:35.211790 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:35.211980 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:35.215616 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:35.215631 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:35.215652 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:35.215705 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:35.219820 (+ 4115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:35.220518 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:35.220525 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:35.220527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:35.220676 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:35.221172 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:35.221176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:35.221178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:35.221676 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:35.221850 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:35.222416 (+ 566us) server_negotiation.cc:299] Negotiation successful
0929 21:02:35.222922 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":167,"server-negotiator.queue_time_us":633,"thread_start_us":408,"threads_started":1}
I20260929 21:02:35.223897 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:35.203659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38435 (local address 127.0.0.1:38110)
0929 21:02:35.204829 (+ 1170us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:35.204917 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0929 21:02:35.206613 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:02:35.207411 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:02:35.207417 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:02:35.208126 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:02:35.208439 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:35.208456 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:35.210019 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:35.210024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:35.211067 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:35.211078 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:35.212016 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:35.212022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:35.212195 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:35.212883 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:02:35.212915 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:02:35.215330 (+ 2415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:02:35.219958 (+ 4628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:35.219967 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:35.219976 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:35.220365 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:35.220789 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:35.220793 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:35.220795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:35.221064 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:35.221807 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:02:35.221812 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:02:35.222062 (+ 250us) client_negotiation.cc:769] Sending connection context
0929 21:02:35.222278 (+ 216us) client_negotiation.cc:240] Negotiation successful
0929 21:02:35.222705 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":963,"thread_start_us":745,"threads_started":1}
I20260929 21:02:35.226769 13882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36597 } has no permanent_uuid. Determining permanent_uuid...
I20260929 21:02:35.227847 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:35.227359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36597 (local address 127.0.0.1:57322)
0929 21:02:35.227560 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:35.227691 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 21:02:35.228096 13832 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36597: connect: Connection refused (error 111)
I20260929 21:02:35.229772 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:35.229298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36597 (local address 127.0.0.1:57330)
0929 21:02:35.229473 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:35.229579 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 21:02:35.229971 13882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36597: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36597: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260929 21:02:35.250209 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:36597
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38435,127.0.97.61:36597,127.0.97.60:39675
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 21:02:35.279321 13882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36597 } attempt: 1
I20260929 21:02:35.280515 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:35.280039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36597 (local address 127.0.0.1:57332)
0929 21:02:35.280271 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:35.280372 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260929 21:02:35.281579 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:35.281225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36597 (local address 127.0.0.1:57336)
0929 21:02:35.281376 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:35.281447 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 21:02:35.281752 13882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36597: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36597: connect: Connection refused (error 111)
I20260929 21:02:35.349988 13882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36597 } attempt: 2
I20260929 21:02:35.351153 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:35.350658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36597 (local address 127.0.0.1:57342)
0929 21:02:35.350910 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:35.351006 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 21:02:35.352144 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:35.351798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36597 (local address 127.0.0.1:57354)
0929 21:02:35.351941 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:35.352021 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 21:02:35.352332 13882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36597: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36597: connect: Connection refused (error 111)
I20260929 21:02:35.443612 13882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36597 } attempt: 3
I20260929 21:02:35.444770 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:35.444338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36597 (local address 127.0.0.1:57364)
0929 21:02:35.444537 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:35.444636 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 21:02:35.445875 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:35.445514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36597 (local address 127.0.0.1:57378)
0929 21:02:35.445683 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:35.445746 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260929 21:02:35.446099 13882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36597: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36597: connect: Connection refused (error 111)
W20260929 21:02:35.576108 13889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:02:35.576449 13889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:02:35.576612 13889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:02:35.583820 13889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:02:35.583966 13889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:02:35.584074 13889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:02:35.584187 13889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:02:35.584273 13889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:02:35.589331 13882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36597 } attempt: 4
I20260929 21:02:35.590510 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:35.589977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36597 (local address 127.0.0.1:57394)
0929 21:02:35.590295 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:35.590380 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260929 21:02:35.591549 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:35.591193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36597 (local address 127.0.0.1:57396)
0929 21:02:35.591354 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:35.591438 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 21:02:35.591728 13882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36597: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36597: connect: Connection refused (error 111)
I20260929 21:02:35.594594 13889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38435,127.0.97.61:36597,127.0.97.60:39675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:36597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:02:35.595916 13889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:02:35.597290 13889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:02:35.608603 13894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:35.608649 13897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:35.608673 13895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:02:35.609539 13889 server_base.cc:1034] running on GCE node
I20260929 21:02:35.610311 13889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:02:35.611701 13889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:02:35.612946 13889 hybrid_clock.cc:648] HybridClock initialized: now 1790715755612897 us; error 40 us; skew 500 ppm
I20260929 21:02:35.613303 13889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:02:35 dist-test-slave-1dms krb5kdc[13802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:02:35.616227 13889 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260929 21:02:35.617703 13889 webserver.cc:460] Webserver started at http://127.0.97.61:40599/ using document root <none> and password file <none>
I20260929 21:02:35.618371 13889 fs_manager.cc:359] Metadata directory not provided
I20260929 21:02:35.618511 13889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:02:35.618808 13889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:02:35.621735 13889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-1/data/instance:
uuid: "b60ff80ce53a4f9aa89cf2b136fdc8a0"
format_stamp: "Formatted at 2026-09-29 21:02:35 on dist-test-slave-1dms"
server_key: "96481e9aa17cee315d3ea0c43df55155"
server_key_iv: "2a252c0e0f96848d015587da5e43d8e7"
server_key_version: "encryptionkey@0"
I20260929 21:02:35.622500 13889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-1/wal/instance:
uuid: "b60ff80ce53a4f9aa89cf2b136fdc8a0"
format_stamp: "Formatted at 2026-09-29 21:02:35 on dist-test-slave-1dms"
server_key: "96481e9aa17cee315d3ea0c43df55155"
server_key_iv: "2a252c0e0f96848d015587da5e43d8e7"
server_key_version: "encryptionkey@0"
I20260929 21:02:35.627573 13889 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 21:02:35.630970 13904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:35.632362 13889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 21:02:35.632551 13889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-1/wal
uuid: "b60ff80ce53a4f9aa89cf2b136fdc8a0"
format_stamp: "Formatted at 2026-09-29 21:02:35 on dist-test-slave-1dms"
server_key: "96481e9aa17cee315d3ea0c43df55155"
server_key_iv: "2a252c0e0f96848d015587da5e43d8e7"
server_key_version: "encryptionkey@0"
I20260929 21:02:35.632761 13889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:02:35.649423 13889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:02:35.653208 13889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:02:35.653482 13889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:02:35.671871 13889 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:36597
I20260929 21:02:35.671901 13955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:36597 every 8 connection(s)
I20260929 21:02:35.673246 13889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-1/data/info.pb
I20260929 21:02:35.676787 13956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:02:35.680014 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 13889
I20260929 21:02:35.680213 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-1/wal/instance
I20260929 21:02:35.680761 388 external_mini_cluster.cc:1428] Setting key bc6234b08b56c41b77148aee17df7b7f
I20260929 21:02:35.685377 13956 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38435 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 21:02:35 dist-test-slave-1dms krb5kdc[13802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260929 21:02:35.707957 13884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:35.688884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38120 (local address 127.0.97.62:38435)
0929 21:02:35.689083 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:35.689090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:35.690306 (+ 1216us) server_negotiation.cc:407] Connection header received
0929 21:02:35.692143 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:35.692152 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:35.692247 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:35.692375 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:35.693996 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:35.695186 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:35.696730 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:35.697553 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:35.697711 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:35.701073 (+ 3362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:35.701081 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:35.701089 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:35.701138 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:35.704763 (+ 3625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:35.705622 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:35.705630 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:35.705637 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:35.705803 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:35.706240 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:35.706246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:35.706250 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:35.706535 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:35.706703 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:35.707396 (+ 693us) server_negotiation.cc:299] Negotiation successful
0929 21:02:35.707619 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 21:02:35.708983 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:35.689314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38435 (local address 127.0.0.1:38120)
0929 21:02:35.690123 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:35.690173 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 21:02:35.691851 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:02:35.692618 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:02:35.692626 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:02:35.693237 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:02:35.693813 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:35.693830 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:35.695357 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:35.695361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:35.696569 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:35.696585 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:35.697772 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:35.697777 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:35.697964 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:35.698693 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:02:35.698715 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:02:35.700789 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:02:35.704941 (+ 4152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:35.704956 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:35.704969 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:35.705494 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:35.705913 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:35.705919 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:35.705923 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:35.706114 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:35.706667 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:02:35.706677 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:02:35.707037 (+ 360us) client_negotiation.cc:769] Sending connection context
0929 21:02:35.707300 (+ 263us) client_negotiation.cc:240] Negotiation successful
0929 21:02:35.707795 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":361,"threads_started":1}
I20260929 21:02:35.710807 13956 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36597 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 21:02:35 dist-test-slave-1dms krb5kdc[13802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 21:02:35.726886 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:35.711609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36597 (local address 127.0.0.1:57412)
0929 21:02:35.711824 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:35.711846 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 21:02:35.711983 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:02:35.712909 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:02:35.712912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:02:35.713248 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:02:35.713379 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:35.713387 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:35.715417 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:35.715420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:35.716232 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:35.716242 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:35.717473 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:35.717477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:35.717633 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:35.718214 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:02:35.718237 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:02:35.720128 (+ 1891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:02:35.724465 (+ 4337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:35.724469 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:35.724472 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:35.724834 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:35.725298 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:35.725301 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:35.725303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:35.725444 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:35.726213 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:02:35.726216 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:02:35.726336 (+ 120us) client_negotiation.cc:769] Sending connection context
0929 21:02:35.726444 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 21:02:35.726637 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 21:02:35.727312 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:35.711673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57412 (local address 127.0.97.61:36597)
0929 21:02:35.712355 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:35.712364 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:35.712398 (+ 34us) server_negotiation.cc:407] Connection header received
0929 21:02:35.712460 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:35.712469 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:35.712703 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:35.712892 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:35.713530 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:35.715260 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:35.716369 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:35.717342 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:35.717480 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:35.720350 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:35.720366 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:35.720387 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:35.720428 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:35.724323 (+ 3895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:35.725013 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:35.725024 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:35.725028 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:35.725167 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:35.725563 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:35.725569 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:35.725572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:35.726086 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:35.726281 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:35.726659 (+ 378us) server_negotiation.cc:299] Negotiation successful
0929 21:02:35.726877 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":483,"thread_start_us":318,"threads_started":1}
I20260929 21:02:35.729795 13956 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39675 } has no permanent_uuid. Determining permanent_uuid...
I20260929 21:02:35.730786 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:35.730368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39675 (local address 127.0.0.1:33244)
0929 21:02:35.730540 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:35.730649 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 21:02:35.731000 13907 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39675: connect: Connection refused (error 111)
I20260929 21:02:35.732404 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:35.732112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39675 (local address 127.0.0.1:33254)
0929 21:02:35.732242 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:35.732316 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 21:02:35.732637 13956 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39675: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39675: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260929 21:02:35.735560 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:39675
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38435,127.0.97.61:36597,127.0.97.60:39675
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 21:02:35.781992 13956 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39675 } attempt: 1
I20260929 21:02:35.783221 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:35.782778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39675 (local address 127.0.0.1:33256)
0929 21:02:35.782971 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:35.783074 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 21:02:35.784349 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:35.784037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39675 (local address 127.0.0.1:33266)
0929 21:02:35.784181 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:35.784248 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 21:02:35.784643 13956 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39675: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39675: connect: Connection refused (error 111)
I20260929 21:02:35.852968 13956 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39675 } attempt: 2
I20260929 21:02:35.854106 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:35.853677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39675 (local address 127.0.0.1:33280)
0929 21:02:35.853868 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:35.853964 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 21:02:35.855046 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:35.854749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39675 (local address 127.0.0.1:33292)
0929 21:02:35.854874 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:35.854938 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 21:02:35.855274 13956 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39675: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39675: connect: Connection refused (error 111)
I20260929 21:02:35.890997 13882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36597 } attempt: 5
Sep 29 21:02:35 dist-test-slave-1dms krb5kdc[13802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 21:02:35.905174 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:35.891782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36597 (local address 127.0.0.1:57418)
0929 21:02:35.892015 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:35.892037 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 21:02:35.892172 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:02:35.892530 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:02:35.892535 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:02:35.893076 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:02:35.893322 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:35.893335 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:35.894291 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:35.894297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:35.895637 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:35.895662 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:35.896484 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:35.896490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:35.896688 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:35.897336 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:02:35.897359 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:02:35.899668 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:02:35.902690 (+ 3022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:35.902696 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:35.902700 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:35.903071 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:35.903438 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:35.903444 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:35.903447 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:35.903583 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:35.904193 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:02:35.904198 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:02:35.904359 (+ 161us) client_negotiation.cc:769] Sending connection context
0929 21:02:35.904501 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 21:02:35.904789 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 21:02:35.905174 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:35.892030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57418 (local address 127.0.97.61:36597)
0929 21:02:35.892234 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:35.892238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:35.892256 (+ 18us) server_negotiation.cc:407] Connection header received
0929 21:02:35.892312 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:35.892315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:35.892374 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:35.892456 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:35.893465 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:35.894133 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:35.895870 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:35.896311 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:35.896407 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:35.899875 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:35.899880 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:35.899883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:35.899914 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:35.902549 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:35.903193 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:35.903200 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:35.903203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:35.903327 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:35.903690 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:35.903696 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:35.903699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:35.904066 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:35.904224 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:35.904612 (+ 388us) server_negotiation.cc:299] Negotiation successful
0929 21:02:35.904814 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 21:02:35.906319 13882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39675 } has no permanent_uuid. Determining permanent_uuid...
I20260929 21:02:35.907286 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:35.906916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39675 (local address 127.0.0.1:33302)
0929 21:02:35.907073 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:35.907150 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 21:02:35.908138 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:35.907830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39675 (local address 127.0.0.1:33316)
0929 21:02:35.907962 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:35.908047 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 21:02:35.908344 13882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39675: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39675: connect: Connection refused (error 111)
I20260929 21:02:35.946483 13956 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39675 } attempt: 3
I20260929 21:02:35.947672 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:35.947215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39675 (local address 127.0.0.1:33324)
0929 21:02:35.947435 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:35.947523 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 21:02:35.948730 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:35.948337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39675 (local address 127.0.0.1:33326)
0929 21:02:35.948544 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:35.948614 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
W20260929 21:02:35.948979 13956 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39675: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39675: connect: Connection refused (error 111)
I20260929 21:02:35.959594 13882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39675 } attempt: 1
I20260929 21:02:35.960594 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:35.960158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39675 (local address 127.0.0.1:33330)
0929 21:02:35.960366 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:35.960445 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 21:02:35.961536 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:35.961222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39675 (local address 127.0.0.1:33344)
0929 21:02:35.961375 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:35.961439 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 21:02:35.961740 13882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39675: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39675: connect: Connection refused (error 111)
I20260929 21:02:36.029999 13882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39675 } attempt: 2
I20260929 21:02:36.031176 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:36.030682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39675 (local address 127.0.0.1:33350)
0929 21:02:36.030932 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:36.031017 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260929 21:02:36.032172 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:36.031847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39675 (local address 127.0.0.1:33352)
0929 21:02:36.031993 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:36.032076 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 21:02:36.032445 13882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39675: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39675: connect: Connection refused (error 111)
W20260929 21:02:36.059832 13963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:02:36.060093 13963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:02:36.060271 13963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:02:36.067282 13963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:02:36.067376 13963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:02:36.067420 13963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:02:36.067499 13963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:02:36.067560 13963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:02:36.077070 13963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38435,127.0.97.61:36597,127.0.97.60:39675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:39675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:02:36.078346 13963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:02:36.079648 13963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:02:36.090974 13968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:36.091033 13969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:36.091009 13971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:02:36.092001 13963 server_base.cc:1034] running on GCE node
I20260929 21:02:36.092192 13956 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39675 } attempt: 4
I20260929 21:02:36.092887 13963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:02:36.093338 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:36.092883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39675 (local address 127.0.0.1:33356)
0929 21:02:36.093113 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:36.093189 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 21:02:36.094272 13963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:02:36.094353 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:36.093968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39675 (local address 127.0.0.1:33370)
0929 21:02:36.094116 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:36.094212 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 21:02:36.094635 13956 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39675: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39675: connect: Connection refused (error 111)
I20260929 21:02:36.095489 13963 hybrid_clock.cc:648] HybridClock initialized: now 1790715756095399 us; error 84 us; skew 500 ppm
I20260929 21:02:36.095817 13963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:02:36 dist-test-slave-1dms krb5kdc[13802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:02:36.098769 13963 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260929 21:02:36.100090 13963 webserver.cc:460] Webserver started at http://127.0.97.60:45227/ using document root <none> and password file <none>
I20260929 21:02:36.100688 13963 fs_manager.cc:359] Metadata directory not provided
I20260929 21:02:36.100780 13963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:02:36.101081 13963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:02:36.103785 13963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-2/data/instance:
uuid: "6e343b3ce1b14328a25486687279c3bc"
format_stamp: "Formatted at 2026-09-29 21:02:36 on dist-test-slave-1dms"
server_key: "2c411fffdfc39abb6e4f3264119e81c9"
server_key_iv: "e2e4eaf19145f257aadd2fb94ff8d6bd"
server_key_version: "encryptionkey@0"
I20260929 21:02:36.104626 13963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-2/wal/instance:
uuid: "6e343b3ce1b14328a25486687279c3bc"
format_stamp: "Formatted at 2026-09-29 21:02:36 on dist-test-slave-1dms"
server_key: "2c411fffdfc39abb6e4f3264119e81c9"
server_key_iv: "e2e4eaf19145f257aadd2fb94ff8d6bd"
server_key_version: "encryptionkey@0"
I20260929 21:02:36.109622 13963 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260929 21:02:36.112816 13978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:36.114173 13963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 21:02:36.114364 13963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-2/wal
uuid: "6e343b3ce1b14328a25486687279c3bc"
format_stamp: "Formatted at 2026-09-29 21:02:36 on dist-test-slave-1dms"
server_key: "2c411fffdfc39abb6e4f3264119e81c9"
server_key_iv: "e2e4eaf19145f257aadd2fb94ff8d6bd"
server_key_version: "encryptionkey@0"
I20260929 21:02:36.114565 13963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:02:36.136047 13963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:02:36.138676 13882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39675 } attempt: 3
I20260929 21:02:36.140127 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:36.139563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39675 (local address 127.0.0.1:33374)
0929 21:02:36.139820 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:36.139923 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 21:02:36.141104 13963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:02:36.141294 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:36.140806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39675 (local address 127.0.0.1:33382)
0929 21:02:36.141063 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:36.141156 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260929 21:02:36.141423 13963 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260929 21:02:36.141520 13882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39675: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39675: connect: Connection refused (error 111)
I20260929 21:02:36.159430 13963 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:39675
I20260929 21:02:36.159454 14029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:39675 every 8 connection(s)
I20260929 21:02:36.160789 13963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-2/data/info.pb
I20260929 21:02:36.164465 14030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:02:36.166122 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 13963
I20260929 21:02:36.166342 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790715556572573-388-0/minicluster-data/master-2/wal/instance
I20260929 21:02:36.166888 388 external_mini_cluster.cc:1428] Setting key 066b35d5f5e9b0914465184e3bb4abe3
I20260929 21:02:36.174544 14030 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38435 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 21:02:36 dist-test-slave-1dms krb5kdc[13802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 29 21:02:36 dist-test-slave-1dms krb5kdc[13802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:02:36.190476 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:36.168957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33384 (local address 127.0.97.60:39675)
0929 21:02:36.169803 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:36.169815 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:36.169854 (+ 39us) server_negotiation.cc:407] Connection header received
0929 21:02:36.171375 (+ 1521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:36.171396 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:36.171802 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:36.172144 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:36.173958 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:36.175344 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:36.176781 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:36.177624 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:36.177823 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:36.182483 (+ 4660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:36.182490 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:36.182501 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:36.182533 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:36.186049 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:36.186875 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:36.186887 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:36.186898 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:36.187070 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:36.187435 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:36.187439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:36.187441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:36.188138 (+ 697us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:36.188436 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:36.189175 (+ 739us) server_negotiation.cc:299] Negotiation successful
0929 21:02:36.189552 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":405,"threads_started":1}
I20260929 21:02:36.193575 13884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:36.177009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38122 (local address 127.0.97.62:38435)
0929 21:02:36.177227 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:36.177236 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:36.178142 (+ 906us) server_negotiation.cc:407] Connection header received
0929 21:02:36.178336 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:36.178343 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:36.178426 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:36.178502 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:36.179603 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:36.180756 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:36.182285 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:36.182757 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:36.182846 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:36.186695 (+ 3849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:36.186700 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:36.186703 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:36.186736 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:36.190522 (+ 3786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:36.191347 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:36.191352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:36.191355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:36.191517 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:36.191916 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:36.191922 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:36.191925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:36.192233 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:36.192368 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:36.192913 (+ 545us) server_negotiation.cc:299] Negotiation successful
0929 21:02:36.193150 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 21:02:36.194088 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:36.177038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38435 (local address 127.0.0.1:38122)
0929 21:02:36.177993 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:36.178041 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 21:02:36.178183 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:02:36.178601 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:02:36.178610 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:02:36.179198 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:02:36.179437 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:36.179452 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:36.180946 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:36.180952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:36.182146 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:36.182158 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:36.182940 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:36.182947 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:36.183150 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:36.183725 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:02:36.183749 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:02:36.186513 (+ 2764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:02:36.190667 (+ 4154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:36.190678 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:36.190685 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:36.191224 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:36.191630 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:36.191634 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:36.191638 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:36.191792 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:36.192369 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:02:36.192375 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:02:36.192544 (+ 169us) client_negotiation.cc:769] Sending connection context
0929 21:02:36.192746 (+ 202us) client_negotiation.cc:240] Negotiation successful
0929 21:02:36.193454 (+ 708us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":479,"threads_started":1}
I20260929 21:02:36.195441 14030 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36597 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 21:02:36 dist-test-slave-1dms krb5kdc[13802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 29 21:02:36 dist-test-slave-1dms krb5kdc[13802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:02:36.209301 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:36.196075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36597 (local address 127.0.0.1:57422)
0929 21:02:36.196208 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:36.196225 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 21:02:36.196313 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:02:36.196690 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:02:36.196693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:02:36.197071 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:02:36.197257 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:36.197271 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:36.198606 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:36.198611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:36.199800 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:36.199816 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:36.200572 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:36.200576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:36.200731 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:36.201301 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:02:36.201317 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:02:36.203289 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:02:36.206986 (+ 3697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:36.206990 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:36.206992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:36.207336 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:36.207749 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:36.207754 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:36.207759 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:36.207922 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:36.208473 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:02:36.208480 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:02:36.208648 (+ 168us) client_negotiation.cc:769] Sending connection context
0929 21:02:36.208808 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 21:02:36.209058 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 21:02:36.209705 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:36.196238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57422 (local address 127.0.97.61:36597)
0929 21:02:36.196420 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:36.196425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:36.196444 (+ 19us) server_negotiation.cc:407] Connection header received
0929 21:02:36.196499 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:36.196505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:36.196570 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:36.196668 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:36.197424 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:36.198456 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:36.199938 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:36.200444 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:36.200541 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:36.203467 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:36.203471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:36.203475 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:36.203507 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:36.206858 (+ 3351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:36.207452 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:36.207458 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:36.207461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:36.207628 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:36.208029 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:36.208032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:36.208034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:36.208337 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:36.208497 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:36.208973 (+ 476us) server_negotiation.cc:299] Negotiation successful
0929 21:02:36.209177 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 21:02:36.210248 13884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:36.193948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38126 (local address 127.0.97.62:38435)
0929 21:02:36.194222 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:36.194228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:36.194243 (+ 15us) server_negotiation.cc:407] Connection header received
0929 21:02:36.194283 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:36.194288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:36.194348 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:36.194445 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:36.195938 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:36.197045 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:36.198357 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:36.198960 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:36.199093 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:36.204372 (+ 5279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:36.204378 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:36.204382 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:36.204417 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:36.207514 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:36.208259 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:36.208266 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:36.208268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:36.208396 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:36.208806 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:36.208811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:36.208813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:36.209177 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:36.209495 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:36.209757 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 21:02:36.209962 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260929 21:02:36.210592 14030 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39675 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 21:02:36 dist-test-slave-1dms krb5kdc[13802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 29 21:02:36 dist-test-slave-1dms krb5kdc[13802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 21:02:36.224359 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:36.211393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33396 (local address 127.0.97.60:39675)
0929 21:02:36.211554 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:36.211558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:36.211594 (+ 36us) server_negotiation.cc:407] Connection header received
0929 21:02:36.211996 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:36.212001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:36.212072 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:36.212169 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:36.213050 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:36.213782 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:36.215107 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:36.215506 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:36.215594 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:36.219055 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:36.219062 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:36.219069 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:36.219106 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:36.221819 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:36.222649 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:36.222653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:36.222655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:36.222769 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:36.223210 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:36.223214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:36.223217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:36.223469 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:36.223574 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:36.223981 (+ 407us) server_negotiation.cc:299] Negotiation successful
0929 21:02:36.224128 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 21:02:36.224752 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:36.211216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39675 (local address 127.0.0.1:33396)
0929 21:02:36.211381 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:36.211399 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 21:02:36.211876 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:02:36.212199 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:02:36.212230 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:02:36.212691 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:02:36.212913 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:36.212929 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:36.213907 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:36.213910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:36.214970 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:36.214984 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:36.215661 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:36.215668 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:36.215867 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:36.216479 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:02:36.216501 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:02:36.218886 (+ 2385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:02:36.221972 (+ 3086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:36.221976 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:36.221978 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:36.222516 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:36.222888 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:36.222894 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:36.222898 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:36.223098 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:36.223596 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:02:36.223603 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:02:36.223749 (+ 146us) client_negotiation.cc:769] Sending connection context
0929 21:02:36.223916 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 21:02:36.224324 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":76}
I20260929 21:02:36.227258 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:36.211744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57432 (local address 127.0.97.61:36597)
0929 21:02:36.211890 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:36.211894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:36.211942 (+ 48us) server_negotiation.cc:407] Connection header received
0929 21:02:36.212083 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:36.212087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:36.212156 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:36.212240 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:36.213618 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:36.214399 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:36.215982 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:36.216353 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:36.216441 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:36.221365 (+ 4924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:36.221370 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:36.221373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:36.221401 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:36.224506 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:36.225290 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:36.225296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:36.225299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:36.225433 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:36.225851 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:36.225854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:36.225856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:36.226175 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:36.226466 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:36.226737 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 21:02:36.226940 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 21:02:36.230885 14030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e343b3ce1b14328a25486687279c3bc: Bootstrap starting.
I20260929 21:02:36.231885 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 21:02:36.234115 14030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e343b3ce1b14328a25486687279c3bc: Neither blocks nor log segments found. Creating new log.
I20260929 21:02:36.235126 14030 log.cc:824] T 00000000000000000000000000000000 P 6e343b3ce1b14328a25486687279c3bc: Log is configured to *not* fsync() on all Append() calls
I20260929 21:02:36.238629 14030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e343b3ce1b14328a25486687279c3bc: No bootstrap required, opened a new log
I20260929 21:02:36.249740 14030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e343b3ce1b14328a25486687279c3bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eb9d6542efb47588bc0246f1e9c7c4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38435 } } peers { permanent_uuid: "b60ff80ce53a4f9aa89cf2b136fdc8a0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36597 } } peers { permanent_uuid: "6e343b3ce1b14328a25486687279c3bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39675 } }
I20260929 21:02:36.250204 14030 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e343b3ce1b14328a25486687279c3bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:02:36.250316 14030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e343b3ce1b14328a25486687279c3bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e343b3ce1b14328a25486687279c3bc, State: Initialized, Role: FOLLOWER
I20260929 21:02:36.251155 14030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e343b3ce1b14328a25486687279c3bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eb9d6542efb47588bc0246f1e9c7c4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38435 } } peers { permanent_uuid: "b60ff80ce53a4f9aa89cf2b136fdc8a0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36597 } } peers { permanent_uuid: "6e343b3ce1b14328a25486687279c3bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39675 } }
I20260929 21:02:36.255985 14043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e343b3ce1b14328a25486687279c3bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eb9d6542efb47588bc0246f1e9c7c4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38435 } } peers { permanent_uuid: "b60ff80ce53a4f9aa89cf2b136fdc8a0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36597 } } peers { permanent_uuid: "6e343b3ce1b14328a25486687279c3bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39675 } } }
I20260929 21:02:36.256414 14043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e343b3ce1b14328a25486687279c3bc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 21:02:36.257539 14030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e343b3ce1b14328a25486687279c3bc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:02:36.258816 13884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:36.236811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38134 (local address 127.0.97.62:38435)
0929 21:02:36.237026 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:36.237032 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:36.238915 (+ 1883us) server_negotiation.cc:407] Connection header received
0929 21:02:36.239190 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:36.239195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:36.239280 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:36.239356 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:36.242406 (+ 3050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:36.243517 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:36.246127 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:36.246804 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:36.246903 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:36.251024 (+ 4121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:36.251030 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:36.251035 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:36.251074 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:36.254898 (+ 3824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:36.256131 (+ 1233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:36.256136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:36.256139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:36.256554 (+ 415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:36.257064 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:36.257070 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:36.257072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:36.257411 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:36.257682 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:36.258213 (+ 531us) server_negotiation.cc:299] Negotiation successful
0929 21:02:36.258440 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 21:02:36.259188 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:36.237956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33410 (local address 127.0.97.60:39675)
0929 21:02:36.238148 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:36.238154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:36.239171 (+ 1017us) server_negotiation.cc:407] Connection header received
0929 21:02:36.239402 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:36.239407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:36.239490 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:36.239564 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:36.242423 (+ 2859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:36.243543 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:36.246127 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:36.246877 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:36.246962 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:36.251024 (+ 4062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:36.251030 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:36.251035 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:36.251072 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:36.254881 (+ 3809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:36.256092 (+ 1211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:36.256100 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:36.256105 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:36.256334 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:36.257202 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:36.257207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:36.257211 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:36.257556 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:36.257827 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:36.258469 (+ 642us) server_negotiation.cc:299] Negotiation successful
0929 21:02:36.258701 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 21:02:36.259295 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:36.236982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57438 (local address 127.0.97.61:36597)
0929 21:02:36.239063 (+ 2081us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:36.239070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:36.239091 (+ 21us) server_negotiation.cc:407] Connection header received
0929 21:02:36.239154 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:36.239160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:36.239237 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:36.240367 (+ 1130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:36.241865 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:36.242957 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:36.245198 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:36.245870 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:36.245990 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:36.249849 (+ 3859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:36.249855 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:36.249859 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:36.249898 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:36.254483 (+ 4585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:36.255681 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:36.255687 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:36.255692 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:36.255864 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:36.256883 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:36.256888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:36.256892 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:36.257309 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:36.257563 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:36.258707 (+ 1144us) server_negotiation.cc:299] Negotiation successful
0929 21:02:36.258964 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1957}
W20260929 21:02:36.268978 14054 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6e343b3ce1b14328a25486687279c3bc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 21:02:36.269186 14054 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 21:02:36.266251 13963 master_runner.cc:186] Error getting master registration for 127.0.97.62:38435: OK, instance_id { permanent_uuid: "6eb9d6542efb47588bc0246f1e9c7c4c" instance_seqno: 1790715755183656 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 21:02:36.276226 13963 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 21:02:36.318732 13882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39675 } attempt: 4
Sep 29 21:02:36 dist-test-slave-1dms krb5kdc[13802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 21:02:36.333137 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:36.319424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33412 (local address 127.0.97.60:39675)
0929 21:02:36.319588 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:36.319596 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:36.319635 (+ 39us) server_negotiation.cc:407] Connection header received
0929 21:02:36.319832 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:36.319838 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:36.319910 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:36.320011 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:36.320790 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:36.321624 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:36.322801 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:36.323279 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:36.323415 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:36.326795 (+ 3380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:36.326802 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:36.326807 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:36.326847 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:36.330400 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:36.331170 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:36.331175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:36.331177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:36.331324 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:36.331721 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:36.331724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:36.331726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:36.331986 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:36.332101 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:36.332568 (+ 467us) server_negotiation.cc:299] Negotiation successful
0929 21:02:36.332767 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 21:02:36.333156 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:36.319350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39675 (local address 127.0.0.1:33412)
0929 21:02:36.319552 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:36.319572 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 21:02:36.319708 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:02:36.320034 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:02:36.320038 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:02:36.320454 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:02:36.320649 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:36.320658 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:36.321771 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:36.321777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:36.322661 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:36.322673 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:36.323425 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:36.323430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:36.323623 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:36.324277 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:02:36.324294 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:02:36.326611 (+ 2317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:02:36.330531 (+ 3920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:36.330537 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:36.330541 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:36.331050 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:36.331431 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:36.331438 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:36.331444 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:36.331616 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:36.332106 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:02:36.332111 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:02:36.332275 (+ 164us) client_negotiation.cc:769] Sending connection context
0929 21:02:36.332428 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 21:02:36.332727 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 21:02:36.337899 13882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eb9d6542efb47588bc0246f1e9c7c4c: Bootstrap starting.
I20260929 21:02:36.341077 13882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6eb9d6542efb47588bc0246f1e9c7c4c: Neither blocks nor log segments found. Creating new log.
I20260929 21:02:36.342245 13882 log.cc:824] T 00000000000000000000000000000000 P 6eb9d6542efb47588bc0246f1e9c7c4c: Log is configured to *not* fsync() on all Append() calls
I20260929 21:02:36.346150 13882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eb9d6542efb47588bc0246f1e9c7c4c: No bootstrap required, opened a new log
I20260929 21:02:36.352887 13882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6eb9d6542efb47588bc0246f1e9c7c4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eb9d6542efb47588bc0246f1e9c7c4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38435 } } peers { permanent_uuid: "b60ff80ce53a4f9aa89cf2b136fdc8a0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36597 } } peers { permanent_uuid: "6e343b3ce1b14328a25486687279c3bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39675 } }
I20260929 21:02:36.353222 13882 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6eb9d6542efb47588bc0246f1e9c7c4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:02:36.353348 13882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6eb9d6542efb47588bc0246f1e9c7c4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6eb9d6542efb47588bc0246f1e9c7c4c, State: Initialized, Role: FOLLOWER
I20260929 21:02:36.353845 13882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6eb9d6542efb47588bc0246f1e9c7c4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eb9d6542efb47588bc0246f1e9c7c4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38435 } } peers { permanent_uuid: "b60ff80ce53a4f9aa89cf2b136fdc8a0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36597 } } peers { permanent_uuid: "6e343b3ce1b14328a25486687279c3bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39675 } }
I20260929 21:02:36.354782 14057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eb9d6542efb47588bc0246f1e9c7c4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eb9d6542efb47588bc0246f1e9c7c4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38435 } } peers { permanent_uuid: "b60ff80ce53a4f9aa89cf2b136fdc8a0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36597 } } peers { permanent_uuid: "6e343b3ce1b14328a25486687279c3bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39675 } } }
I20260929 21:02:36.355175 14057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eb9d6542efb47588bc0246f1e9c7c4c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 21:02:36.355865 13882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6eb9d6542efb47588bc0246f1e9c7c4c [sys.catalog]: configured and running, proceeding with master startup.
W20260929 21:02:36.366667 14068 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6eb9d6542efb47588bc0246f1e9c7c4c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 21:02:36.366793 14068 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 21:02:36.366088 13815 master_runner.cc:186] Error getting master registration for 127.0.97.61:36597: OK, instance_id { permanent_uuid: "b60ff80ce53a4f9aa89cf2b136fdc8a0" instance_seqno: 1790715755666579 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 21:02:36.377827 13815 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 21:02:36.393533 13984 tablet.cc:2369] T 00000000000000000000000000000000 P 6e343b3ce1b14328a25486687279c3bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 21:02:36.393905 13956 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39675 } attempt: 5
Sep 29 21:02:36 dist-test-slave-1dms krb5kdc[13802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 21:02:36.410951 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:36.394418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39675 (local address 127.0.0.1:33422)
0929 21:02:36.394693 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:36.394712 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 21:02:36.394821 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:02:36.395174 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:02:36.395179 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:02:36.395717 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:02:36.395928 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:36.395941 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:36.397398 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:36.397404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:36.398873 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:36.398888 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:36.399873 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:36.399879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:36.400082 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:36.400790 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:02:36.400815 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:02:36.403708 (+ 2893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:02:36.408009 (+ 4301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:36.408015 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:36.408020 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:36.408533 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:36.409035 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:36.409040 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:36.409044 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:36.409225 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:36.409938 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:02:36.409943 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:02:36.410098 (+ 155us) client_negotiation.cc:769] Sending connection context
0929 21:02:36.410256 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 21:02:36.410548 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136}
I20260929 21:02:36.411975 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:36.394505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33422 (local address 127.0.97.60:39675)
0929 21:02:36.394697 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:36.394704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:36.394829 (+ 125us) server_negotiation.cc:407] Connection header received
0929 21:02:36.394929 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:36.394934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:36.395010 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:36.395120 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:36.396088 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:36.397244 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:36.399027 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:36.399720 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:36.399852 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:36.403894 (+ 4042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:36.403901 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:36.403906 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:36.403943 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:36.407871 (+ 3928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:36.408712 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:36.408717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:36.408721 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:36.408895 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:36.409358 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:36.409364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:36.409367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:36.409732 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:36.409865 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:36.411313 (+ 1448us) server_negotiation.cc:299] Negotiation successful
0929 21:02:36.411579 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 21:02:36.416461 14043 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e343b3ce1b14328a25486687279c3bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 21:02:36.416772 14043 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e343b3ce1b14328a25486687279c3bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eb9d6542efb47588bc0246f1e9c7c4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38435 } } peers { permanent_uuid: "b60ff80ce53a4f9aa89cf2b136fdc8a0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36597 } } peers { permanent_uuid: "6e343b3ce1b14328a25486687279c3bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39675 } }
I20260929 21:02:36.418452 14043 leader_election.cc:288] T 00000000000000000000000000000000 P 6e343b3ce1b14328a25486687279c3bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6eb9d6542efb47588bc0246f1e9c7c4c (127.0.97.62:38435), b60ff80ce53a4f9aa89cf2b136fdc8a0 (127.0.97.61:36597)
I20260929 21:02:36.418855 13956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b60ff80ce53a4f9aa89cf2b136fdc8a0: Bootstrap starting.
I20260929 21:02:36.420387 13857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6e343b3ce1b14328a25486687279c3bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6eb9d6542efb47588bc0246f1e9c7c4c" is_pre_election: true
I20260929 21:02:36.421051 13857 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6eb9d6542efb47588bc0246f1e9c7c4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e343b3ce1b14328a25486687279c3bc in term 0.
I20260929 21:02:36.421762 13981 leader_election.cc:302] T 00000000000000000000000000000000 P 6e343b3ce1b14328a25486687279c3bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e343b3ce1b14328a25486687279c3bc, 6eb9d6542efb47588bc0246f1e9c7c4c; no voters:
I20260929 21:02:36.422226 14043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e343b3ce1b14328a25486687279c3bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 21:02:36.422370 14043 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e343b3ce1b14328a25486687279c3bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 21:02:36.422479 14043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e343b3ce1b14328a25486687279c3bc [term 0 FOLLOWER]: Advancing to term 1
W20260929 21:02:36.423372 13836 tablet.cc:2369] T 00000000000000000000000000000000 P 6eb9d6542efb47588bc0246f1e9c7c4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 21:02:36.423908 13956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b60ff80ce53a4f9aa89cf2b136fdc8a0: Neither blocks nor log segments found. Creating new log.
I20260929 21:02:36.425520 13956 log.cc:824] T 00000000000000000000000000000000 P b60ff80ce53a4f9aa89cf2b136fdc8a0: Log is configured to *not* fsync() on all Append() calls
I20260929 21:02:36.425966 14043 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e343b3ce1b14328a25486687279c3bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eb9d6542efb47588bc0246f1e9c7c4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38435 } } peers { permanent_uuid: "b60ff80ce53a4f9aa89cf2b136fdc8a0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36597 } } peers { permanent_uuid: "6e343b3ce1b14328a25486687279c3bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39675 } }
I20260929 21:02:36.427205 13857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6e343b3ce1b14328a25486687279c3bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6eb9d6542efb47588bc0246f1e9c7c4c"
I20260929 21:02:36.427428 13857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6eb9d6542efb47588bc0246f1e9c7c4c [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:02:36.430795 13857 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6eb9d6542efb47588bc0246f1e9c7c4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e343b3ce1b14328a25486687279c3bc in term 1.
I20260929 21:02:36.431277 13981 leader_election.cc:302] T 00000000000000000000000000000000 P 6e343b3ce1b14328a25486687279c3bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e343b3ce1b14328a25486687279c3bc, 6eb9d6542efb47588bc0246f1e9c7c4c; no voters:
I20260929 21:02:36.437263 14043 leader_election.cc:288] T 00000000000000000000000000000000 P 6e343b3ce1b14328a25486687279c3bc [CANDIDATE]: Term 1 election: Requested vote from peers 6eb9d6542efb47588bc0246f1e9c7c4c (127.0.97.62:38435), b60ff80ce53a4f9aa89cf2b136fdc8a0 (127.0.97.61:36597)
I20260929 21:02:36.438329 14043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e343b3ce1b14328a25486687279c3bc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:02:36.439191 14043 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e343b3ce1b14328a25486687279c3bc [term 1 LEADER]: Becoming Leader. State: Replica: 6e343b3ce1b14328a25486687279c3bc, State: Running, Role: LEADER
I20260929 21:02:36.442448 14043 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e343b3ce1b14328a25486687279c3bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eb9d6542efb47588bc0246f1e9c7c4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38435 } } peers { permanent_uuid: "b60ff80ce53a4f9aa89cf2b136fdc8a0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36597 } } peers { permanent_uuid: "6e343b3ce1b14328a25486687279c3bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39675 } }
I20260929 21:02:36.453387 14071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e343b3ce1b14328a25486687279c3bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e343b3ce1b14328a25486687279c3bc. Latest consensus state: current_term: 1 leader_uuid: "6e343b3ce1b14328a25486687279c3bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eb9d6542efb47588bc0246f1e9c7c4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38435 } } peers { permanent_uuid: "b60ff80ce53a4f9aa89cf2b136fdc8a0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36597 } } peers { permanent_uuid: "6e343b3ce1b14328a25486687279c3bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39675 } } }
I20260929 21:02:36.454362 14071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e343b3ce1b14328a25486687279c3bc [sys.catalog]: This master's current role is: LEADER
I20260929 21:02:36.458276 13956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b60ff80ce53a4f9aa89cf2b136fdc8a0: No bootstrap required, opened a new log
I20260929 21:02:36.465919 14073 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:02:36.468350 14073 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:02:36.481666 13956 raft_consensus.cc:348] T 00000000000000000000000000000000 P b60ff80ce53a4f9aa89cf2b136fdc8a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eb9d6542efb47588bc0246f1e9c7c4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38435 } } peers { permanent_uuid: "b60ff80ce53a4f9aa89cf2b136fdc8a0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36597 } } peers { permanent_uuid: "6e343b3ce1b14328a25486687279c3bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39675 } }
I20260929 21:02:36.482414 13956 raft_consensus.cc:374] T 00000000000000000000000000000000 P b60ff80ce53a4f9aa89cf2b136fdc8a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:02:36.482616 13956 raft_consensus.cc:729] T 00000000000000000000000000000000 P b60ff80ce53a4f9aa89cf2b136fdc8a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b60ff80ce53a4f9aa89cf2b136fdc8a0, State: Initialized, Role: FOLLOWER
I20260929 21:02:36.483809 13956 consensus_queue.cc:260] T 00000000000000000000000000000000 P b60ff80ce53a4f9aa89cf2b136fdc8a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eb9d6542efb47588bc0246f1e9c7c4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38435 } } peers { permanent_uuid: "b60ff80ce53a4f9aa89cf2b136fdc8a0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36597 } } peers { permanent_uuid: "6e343b3ce1b14328a25486687279c3bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39675 } }
I20260929 21:02:36.485814 13857 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6eb9d6542efb47588bc0246f1e9c7c4c [term 1 FOLLOWER]: Refusing update from remote peer 6e343b3ce1b14328a25486687279c3bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:02:36.487222 14071 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6e343b3ce1b14328a25486687279c3bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6eb9d6542efb47588bc0246f1e9c7c4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:02:36.497371 14075 sys_catalog.cc:455] T 00000000000000000000000000000000 P b60ff80ce53a4f9aa89cf2b136fdc8a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eb9d6542efb47588bc0246f1e9c7c4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38435 } } peers { permanent_uuid: "b60ff80ce53a4f9aa89cf2b136fdc8a0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36597 } } peers { permanent_uuid: "6e343b3ce1b14328a25486687279c3bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39675 } } }
I20260929 21:02:36.497800 14075 sys_catalog.cc:458] T 00000000000000000000000000000000 P b60ff80ce53a4f9aa89cf2b136fdc8a0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 21:02:36.500028 13931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6e343b3ce1b14328a25486687279c3bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b60ff80ce53a4f9aa89cf2b136fdc8a0" is_pre_election: true
I20260929 21:02:36.500370 13929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b60ff80ce53a4f9aa89cf2b136fdc8a0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:02:36.504266 13929 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b60ff80ce53a4f9aa89cf2b136fdc8a0 [term 1 FOLLOWER]: Refusing update from remote peer 6e343b3ce1b14328a25486687279c3bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:02:36.505467 13930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6e343b3ce1b14328a25486687279c3bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b60ff80ce53a4f9aa89cf2b136fdc8a0"
I20260929 21:02:36.505097 14071 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6e343b3ce1b14328a25486687279c3bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b60ff80ce53a4f9aa89cf2b136fdc8a0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:02:36.510633 13930 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b60ff80ce53a4f9aa89cf2b136fdc8a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e343b3ce1b14328a25486687279c3bc in term 1.
I20260929 21:02:36.510975 13956 sys_catalog.cc:564] T 00000000000000000000000000000000 P b60ff80ce53a4f9aa89cf2b136fdc8a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:02:36.528772 14057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eb9d6542efb47588bc0246f1e9c7c4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e343b3ce1b14328a25486687279c3bc. Latest consensus state: current_term: 1 leader_uuid: "6e343b3ce1b14328a25486687279c3bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eb9d6542efb47588bc0246f1e9c7c4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38435 } } peers { permanent_uuid: "b60ff80ce53a4f9aa89cf2b136fdc8a0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36597 } } peers { permanent_uuid: "6e343b3ce1b14328a25486687279c3bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39675 } } }
I20260929 21:02:36.529167 14057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eb9d6542efb47588bc0246f1e9c7c4c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 21:02:36.531574 14074 mvcc.cc:205] Tried to move back new op lower bound from 7334771738549559296 to 7334771738400542720. Current Snapshot: MvccSnapshot[applied={T|T < 7334771738549559296}]
I20260929 21:02:36.534366 14075 sys_catalog.cc:455] T 00000000000000000000000000000000 P b60ff80ce53a4f9aa89cf2b136fdc8a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e343b3ce1b14328a25486687279c3bc. Latest consensus state: current_term: 1 leader_uuid: "6e343b3ce1b14328a25486687279c3bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eb9d6542efb47588bc0246f1e9c7c4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38435 } } peers { permanent_uuid: "b60ff80ce53a4f9aa89cf2b136fdc8a0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36597 } } peers { permanent_uuid: "6e343b3ce1b14328a25486687279c3bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39675 } } }
I20260929 21:02:36.534765 14075 sys_catalog.cc:458] T 00000000000000000000000000000000 P b60ff80ce53a4f9aa89cf2b136fdc8a0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 21:02:36.537505 14079 mvcc.cc:205] Tried to move back new op lower bound from 7334771738549559296 to 7334771738400542720. Current Snapshot: MvccSnapshot[applied={T|T < 7334771738549559296}]
I20260929 21:02:36.542101 14071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e343b3ce1b14328a25486687279c3bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6e343b3ce1b14328a25486687279c3bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eb9d6542efb47588bc0246f1e9c7c4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38435 } } peers { permanent_uuid: "b60ff80ce53a4f9aa89cf2b136fdc8a0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36597 } } peers { permanent_uuid: "6e343b3ce1b14328a25486687279c3bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39675 } } }
I20260929 21:02:36.542342 14071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e343b3ce1b14328a25486687279c3bc [sys.catalog]: This master's current role is: LEADER
I20260929 21:02:36.542699 14071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e343b3ce1b14328a25486687279c3bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6e343b3ce1b14328a25486687279c3bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eb9d6542efb47588bc0246f1e9c7c4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38435 } } peers { permanent_uuid: "b60ff80ce53a4f9aa89cf2b136fdc8a0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36597 } } peers { permanent_uuid: "6e343b3ce1b14328a25486687279c3bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39675 } } }
I20260929 21:02:36.542915 14071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e343b3ce1b14328a25486687279c3bc [sys.catalog]: This master's current role is: LEADER
I20260929 21:02:36.543232 14076 mvcc.cc:205] Tried to move back new op lower bound from 7334771738549559296 to 7334771738400542720. Current Snapshot: MvccSnapshot[applied={T|T < 7334771738549559296}]
I20260929 21:02:36.544764 14073 catalog_manager.cc:1347] Generated new cluster ID: 2e25f57495d64128b1210581595ac22c
I20260929 21:02:36.544905 14073 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:02:36.553728 14057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eb9d6542efb47588bc0246f1e9c7c4c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6e343b3ce1b14328a25486687279c3bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eb9d6542efb47588bc0246f1e9c7c4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38435 } } peers { permanent_uuid: "b60ff80ce53a4f9aa89cf2b136fdc8a0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36597 } } peers { permanent_uuid: "6e343b3ce1b14328a25486687279c3bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39675 } } }
I20260929 21:02:36.553979 14057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eb9d6542efb47588bc0246f1e9c7c4c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 21:02:36.556187 14075 sys_catalog.cc:455] T 00000000000000000000000000000000 P b60ff80ce53a4f9aa89cf2b136fdc8a0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6e343b3ce1b14328a25486687279c3bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eb9d6542efb47588bc0246f1e9c7c4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38435 } } peers { permanent_uuid: "b60ff80ce53a4f9aa89cf2b136fdc8a0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36597 } } peers { permanent_uuid: "6e343b3ce1b14328a25486687279c3bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39675 } } }
I20260929 21:02:36.556430 14075 sys_catalog.cc:458] T 00000000000000000000000000000000 P b60ff80ce53a4f9aa89cf2b136fdc8a0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 21:02:36.575696 14073 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:02:36.577678 14073 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:02:36.589303 14073 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e343b3ce1b14328a25486687279c3bc: Generated new TSK 0
I20260929 21:02:36.590385 14073 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 21:02:36.612110 14093 catalog_manager.cc:1259] Loaded cluster ID: 2e25f57495d64128b1210581595ac22c
I20260929 21:02:36.612221 14093 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b60ff80ce53a4f9aa89cf2b136fdc8a0: loading cluster ID for follower catalog manager: success
I20260929 21:02:36.616426 14093 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b60ff80ce53a4f9aa89cf2b136fdc8a0: acquiring CA information for follower catalog manager: success
I20260929 21:02:36.619760 14093 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b60ff80ce53a4f9aa89cf2b136fdc8a0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 21:02:37.368680 14068 catalog_manager.cc:1259] Loaded cluster ID: 2e25f57495d64128b1210581595ac22c
I20260929 21:02:37.368820 14068 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6eb9d6542efb47588bc0246f1e9c7c4c: loading cluster ID for follower catalog manager: success
I20260929 21:02:37.371373 14068 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6eb9d6542efb47588bc0246f1e9c7c4c: acquiring CA information for follower catalog manager: success
I20260929 21:02:37.373296 14068 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6eb9d6542efb47588bc0246f1e9c7c4c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 21:02:37.721706 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 13815
I20260929 21:02:37.723013 13877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:02:37.832022 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 13815
I20260929 21:02:37.852690 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 13889
I20260929 21:02:37.853801 13951 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:02:37.960114 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 13889
I20260929 21:02:37.981307 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 13963
I20260929 21:02:37.982352 14025 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:02:38.090416 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 13963
2026-09-29T21:02:38Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4661 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 21:02:38 dist-test-slave-1dms krb5kdc[14103](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 21:02:38 dist-test-slave-1dms krb5kdc[14103](info): set up 2 sockets
Sep 29 21:02:38 dist-test-slave-1dms krb5kdc[14103](info): commencing operation
krb5kdc: starting...
W20260929 21:02:39.189303 388 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.004s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 21:02:39 dist-test-slave-1dms krb5kdc[14103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T21:02:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:02:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 21:02:39.413797 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38685
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38685
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:02:39.734790 14116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:02:39.735067 14116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:02:39.735164 14116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:02:39.742228 14116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:02:39.742322 14116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:02:39.742406 14116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:02:39.742502 14116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:02:39.742550 14116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:02:39.752209 14116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:02:39.753532 14116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:02:39.754731 14116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:02:39.766290 14124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:39.766346 14122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:39.766373 14121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:02:39.774514 14116 server_base.cc:1034] running on GCE node
I20260929 21:02:39.775384 14116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:02:39.776942 14116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:02:39.778167 14116 hybrid_clock.cc:648] HybridClock initialized: now 1790715759778119 us; error 57 us; skew 500 ppm
I20260929 21:02:39.778514 14116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:02:39 dist-test-slave-1dms krb5kdc[14103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:02:39.781673 14116 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 21:02:39.783317 14116 webserver.cc:460] Webserver started at http://127.0.97.62:36531/ using document root <none> and password file <none>
I20260929 21:02:39.784037 14116 fs_manager.cc:359] Metadata directory not provided
I20260929 21:02:39.784175 14116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:02:39.784482 14116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:02:39.787446 14116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "df8708bbdc374816a608ca76898950a6"
format_stamp: "Formatted at 2026-09-29 21:02:39 on dist-test-slave-1dms"
server_key: "401a8d91f29550ad1553ad7296d6dadc"
server_key_iv: "604418835f130f556f6b0e0b3e0adedd"
server_key_version: "encryptionkey@0"
I20260929 21:02:39.788177 14116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "df8708bbdc374816a608ca76898950a6"
format_stamp: "Formatted at 2026-09-29 21:02:39 on dist-test-slave-1dms"
server_key: "401a8d91f29550ad1553ad7296d6dadc"
server_key_iv: "604418835f130f556f6b0e0b3e0adedd"
server_key_version: "encryptionkey@0"
I20260929 21:02:39.793239 14116 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260929 21:02:39.796476 14131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:39.797735 14116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:02:39.797926 14116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "df8708bbdc374816a608ca76898950a6"
format_stamp: "Formatted at 2026-09-29 21:02:39 on dist-test-slave-1dms"
server_key: "401a8d91f29550ad1553ad7296d6dadc"
server_key_iv: "604418835f130f556f6b0e0b3e0adedd"
server_key_version: "encryptionkey@0"
I20260929 21:02:39.798120 14116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:02:39.821415 14116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:02:39.825800 14116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:02:39.826089 14116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:02:39.844086 14116 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38685
I20260929 21:02:39.844193 14182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38685 every 8 connection(s)
I20260929 21:02:39.845463 14116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:02:39.849314 14183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:02:39.854661 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 14116
I20260929 21:02:39.854892 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:02:39.855505 388 external_mini_cluster.cc:1428] Setting key 6a30a7bbd8bf7a873f798758bcfcf0f6
I20260929 21:02:39.860613 14183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df8708bbdc374816a608ca76898950a6: Bootstrap starting.
I20260929 21:02:39.864286 14183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df8708bbdc374816a608ca76898950a6: Neither blocks nor log segments found. Creating new log.
I20260929 21:02:39.865422 14183 log.cc:824] T 00000000000000000000000000000000 P df8708bbdc374816a608ca76898950a6: Log is configured to *not* fsync() on all Append() calls
I20260929 21:02:39.868426 14183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df8708bbdc374816a608ca76898950a6: No bootstrap required, opened a new log
Sep 29 21:02:39 dist-test-slave-1dms krb5kdc[14103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:02:39.878124 14183 raft_consensus.cc:348] T 00000000000000000000000000000000 P df8708bbdc374816a608ca76898950a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df8708bbdc374816a608ca76898950a6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38685 } }
I20260929 21:02:39.878643 14183 raft_consensus.cc:374] T 00000000000000000000000000000000 P df8708bbdc374816a608ca76898950a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:02:39.878760 14183 raft_consensus.cc:729] T 00000000000000000000000000000000 P df8708bbdc374816a608ca76898950a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8708bbdc374816a608ca76898950a6, State: Initialized, Role: FOLLOWER
I20260929 21:02:39.879412 14183 consensus_queue.cc:260] T 00000000000000000000000000000000 P df8708bbdc374816a608ca76898950a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df8708bbdc374816a608ca76898950a6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38685 } }
I20260929 21:02:39.879675 14183 raft_consensus.cc:388] T 00000000000000000000000000000000 P df8708bbdc374816a608ca76898950a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:02:39.879819 14183 raft_consensus.cc:482] T 00000000000000000000000000000000 P df8708bbdc374816a608ca76898950a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:02:39.880005 14183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df8708bbdc374816a608ca76898950a6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:02:39.880232 14186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:39.857359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57388 (local address 127.0.97.62:38685)
0929 21:02:39.858169 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:39.858186 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:39.858236 (+ 50us) server_negotiation.cc:407] Connection header received
0929 21:02:39.860575 (+ 2339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:39.860610 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:39.861140 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:39.861563 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:39.862832 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:39.864819 (+ 1987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:39.866443 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:39.867386 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:39.867568 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:39.871578 (+ 4010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:39.871590 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:39.871614 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:39.871656 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:39.875618 (+ 3962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:39.876402 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:39.876414 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:39.876427 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:39.876641 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:39.877049 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:39.877054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:39.877057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:39.877744 (+ 687us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:39.878012 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:39.878364 (+ 352us) server_negotiation.cc:299] Negotiation successful
0929 21:02:39.878868 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":345,"threads_started":1}
I20260929 21:02:39.882974 14183 raft_consensus.cc:504] T 00000000000000000000000000000000 P df8708bbdc374816a608ca76898950a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df8708bbdc374816a608ca76898950a6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38685 } }
I20260929 21:02:39.883505 14183 leader_election.cc:302] T 00000000000000000000000000000000 P df8708bbdc374816a608ca76898950a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df8708bbdc374816a608ca76898950a6; no voters:
I20260929 21:02:39.884152 14183 leader_election.cc:288] T 00000000000000000000000000000000 P df8708bbdc374816a608ca76898950a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 21:02:39.884298 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 21:02:39.884284 14188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df8708bbdc374816a608ca76898950a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:02:39.886217 14183 sys_catalog.cc:564] T 00000000000000000000000000000000 P df8708bbdc374816a608ca76898950a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:02:39.889164 14188 raft_consensus.cc:686] T 00000000000000000000000000000000 P df8708bbdc374816a608ca76898950a6 [term 1 LEADER]: Becoming Leader. State: Replica: df8708bbdc374816a608ca76898950a6, State: Running, Role: LEADER
I20260929 21:02:39.889822 14188 consensus_queue.cc:237] T 00000000000000000000000000000000 P df8708bbdc374816a608ca76898950a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df8708bbdc374816a608ca76898950a6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38685 } }
I20260929 21:02:39.892319 14189 sys_catalog.cc:455] T 00000000000000000000000000000000 P df8708bbdc374816a608ca76898950a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df8708bbdc374816a608ca76898950a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df8708bbdc374816a608ca76898950a6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38685 } } }
I20260929 21:02:39.892678 14189 sys_catalog.cc:458] T 00000000000000000000000000000000 P df8708bbdc374816a608ca76898950a6 [sys.catalog]: This master's current role is: LEADER
I20260929 21:02:39.892962 14190 sys_catalog.cc:455] T 00000000000000000000000000000000 P df8708bbdc374816a608ca76898950a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df8708bbdc374816a608ca76898950a6. Latest consensus state: current_term: 1 leader_uuid: "df8708bbdc374816a608ca76898950a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df8708bbdc374816a608ca76898950a6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38685 } } }
I20260929 21:02:39.893414 14190 sys_catalog.cc:458] T 00000000000000000000000000000000 P df8708bbdc374816a608ca76898950a6 [sys.catalog]: This master's current role is: LEADER
W20260929 21:02:39.899734 14204 catalog_manager.cc:1558] T 00000000000000000000000000000000 P df8708bbdc374816a608ca76898950a6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 21:02:39.899883 14204 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 21:02:39.900017 14199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:02:39.901810 14199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:02:39.909085 14199 catalog_manager.cc:1347] Generated new cluster ID: d128dee6f4f64d63883fc60c9a46f360
I20260929 21:02:39.909196 14199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:02:39.940362 14199 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:02:39.941684 14199 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:02:39.948796 14199 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df8708bbdc374816a608ca76898950a6: Generated new TSK 0
I20260929 21:02:39.949824 14199 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 21:02:39.997782 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 14116
I20260929 21:02:39.998939 14178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:02:40.096211 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 14116
2026-09-29T21:02:40Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2000 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 21:02:40 dist-test-slave-1dms krb5kdc[14211](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 21:02:40 dist-test-slave-1dms krb5kdc[14211](info): set up 2 sockets
Sep 29 21:02:40 dist-test-slave-1dms krb5kdc[14211](info): commencing operation
krb5kdc: starting...
W20260929 21:02:41.197257 388 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 21:02:41 dist-test-slave-1dms krb5kdc[14211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T21:02:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:02:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 21:02:41.416807 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39983
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39983
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:02:41.736155 14224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:02:41.736493 14224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:02:41.736658 14224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:02:41.743853 14224 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:02:41.743981 14224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:02:41.744091 14224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:02:41.744186 14224 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:02:41.744252 14224 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:02:41.753935 14224 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:02:41.755177 14224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:02:41.756438 14224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:02:41.767908 14229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:41.767935 14232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:41.767853 14230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:02:41.768702 14224 server_base.cc:1034] running on GCE node
I20260929 21:02:41.769461 14224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:02:41.770831 14224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:02:41.772089 14224 hybrid_clock.cc:648] HybridClock initialized: now 1790715761772040 us; error 48 us; skew 500 ppm
I20260929 21:02:41.772465 14224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:02:41 dist-test-slave-1dms krb5kdc[14211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:02:41.775555 14224 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 21:02:41.777070 14224 webserver.cc:460] Webserver started at http://127.0.97.62:39115/ using document root <none> and password file <none>
I20260929 21:02:41.777736 14224 fs_manager.cc:359] Metadata directory not provided
I20260929 21:02:41.777874 14224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:02:41.778172 14224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:02:41.781126 14224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "bda88db6437c46fda958bcb092445d16"
format_stamp: "Formatted at 2026-09-29 21:02:41 on dist-test-slave-1dms"
server_key: "1eff2d0bff18c77852696b1f5d453896"
server_key_iv: "3593af8c53c54a3d5497dd17e8de5696"
server_key_version: "encryptionkey@0"
I20260929 21:02:41.781883 14224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "bda88db6437c46fda958bcb092445d16"
format_stamp: "Formatted at 2026-09-29 21:02:41 on dist-test-slave-1dms"
server_key: "1eff2d0bff18c77852696b1f5d453896"
server_key_iv: "3593af8c53c54a3d5497dd17e8de5696"
server_key_version: "encryptionkey@0"
I20260929 21:02:41.786972 14224 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 21:02:41.790246 14239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:41.791500 14224 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 21:02:41.791667 14224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "bda88db6437c46fda958bcb092445d16"
format_stamp: "Formatted at 2026-09-29 21:02:41 on dist-test-slave-1dms"
server_key: "1eff2d0bff18c77852696b1f5d453896"
server_key_iv: "3593af8c53c54a3d5497dd17e8de5696"
server_key_version: "encryptionkey@0"
I20260929 21:02:41.791853 14224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:02:41.807997 14224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:02:41.812108 14224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:02:41.812391 14224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:02:41.830206 14224 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39983
I20260929 21:02:41.830243 14290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39983 every 8 connection(s)
I20260929 21:02:41.831518 14224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:02:41.835009 14291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:02:41.837327 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 14224
I20260929 21:02:41.837541 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:02:41.838332 388 external_mini_cluster.cc:1428] Setting key 34d50721d532ed5278434135776f12bc
I20260929 21:02:41.847214 14291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16: Bootstrap starting.
I20260929 21:02:41.850824 14291 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16: Neither blocks nor log segments found. Creating new log.
I20260929 21:02:41.852126 14291 log.cc:824] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16: Log is configured to *not* fsync() on all Append() calls
Sep 29 21:02:41 dist-test-slave-1dms krb5kdc[14211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:02:41.854874 14291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16: No bootstrap required, opened a new log
I20260929 21:02:41.861151 14294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:41.840392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45382 (local address 127.0.97.62:39983)
0929 21:02:41.841227 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:41.841238 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:41.841283 (+ 45us) server_negotiation.cc:407] Connection header received
0929 21:02:41.842880 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:41.842901 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:41.843263 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:41.843594 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:41.845292 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:41.846551 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:41.848320 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:41.849031 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:41.849166 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:41.854043 (+ 4877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:41.854051 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:41.854064 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:41.854096 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:41.857224 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:41.857892 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:41.857900 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:41.857915 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:41.858086 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:41.858461 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:41.858470 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:41.858472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:41.858991 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:41.859302 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:41.859737 (+ 435us) server_negotiation.cc:299] Negotiation successful
0929 21:02:41.860135 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":407,"threads_started":1}
I20260929 21:02:41.862443 14291 raft_consensus.cc:348] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda88db6437c46fda958bcb092445d16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39983 } }
I20260929 21:02:41.862787 14291 raft_consensus.cc:374] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:02:41.862895 14291 raft_consensus.cc:729] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bda88db6437c46fda958bcb092445d16, State: Initialized, Role: FOLLOWER
I20260929 21:02:41.863405 14291 consensus_queue.cc:260] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda88db6437c46fda958bcb092445d16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39983 } }
I20260929 21:02:41.863641 14291 raft_consensus.cc:388] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:02:41.863796 14291 raft_consensus.cc:482] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:02:41.863963 14291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:02:41.866051 14291 raft_consensus.cc:504] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda88db6437c46fda958bcb092445d16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39983 } }
I20260929 21:02:41.866591 14291 leader_election.cc:302] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bda88db6437c46fda958bcb092445d16; no voters:
I20260929 21:02:41.867218 14291 leader_election.cc:288] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 21:02:41.867324 14296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:02:41.867947 14296 raft_consensus.cc:686] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [term 1 LEADER]: Becoming Leader. State: Replica: bda88db6437c46fda958bcb092445d16, State: Running, Role: LEADER
I20260929 21:02:41.868523 14296 consensus_queue.cc:237] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda88db6437c46fda958bcb092445d16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39983 } }
I20260929 21:02:41.869369 14291 sys_catalog.cc:564] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:02:41.871783 14297 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bda88db6437c46fda958bcb092445d16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda88db6437c46fda958bcb092445d16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39983 } } }
I20260929 21:02:41.872156 14297 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [sys.catalog]: This master's current role is: LEADER
I20260929 21:02:41.874833 14298 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bda88db6437c46fda958bcb092445d16. Latest consensus state: current_term: 1 leader_uuid: "bda88db6437c46fda958bcb092445d16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda88db6437c46fda958bcb092445d16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39983 } } }
I20260929 21:02:41.875253 14298 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [sys.catalog]: This master's current role is: LEADER
I20260929 21:02:41.876207 14304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:02:41.880687 14304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:02:41.888926 14304 catalog_manager.cc:1347] Generated new cluster ID: 638bfd7a910c45d59fcd4bff57f4e24d
I20260929 21:02:41.889042 14304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:02:41.904670 14304 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:02:41.906416 14304 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:02:41.923182 14304 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16: Generated new TSK 0
I20260929 21:02:41.924068 14304 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 21:02:41.976188 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39983
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:02:42.297350 14317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:02:42.297611 14317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:02:42.297793 14317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:02:42.304952 14317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:02:42.305073 14317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:02:42.305236 14317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 21:02:42.314965 14317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:02:42.316602 14317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:02:42.317905 14317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:02:42.330699 14322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:42.330765 14325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:42.330739 14323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:02:42.331895 14317 server_base.cc:1034] running on GCE node
I20260929 21:02:42.332468 14317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:02:42.333395 14317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:02:42.334584 14317 hybrid_clock.cc:648] HybridClock initialized: now 1790715762334539 us; error 43 us; skew 500 ppm
I20260929 21:02:42.334848 14317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:02:42 dist-test-slave-1dms krb5kdc[14211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:02:42.337863 14317 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 21:02:42.339310 14317 webserver.cc:460] Webserver started at http://127.0.97.1:42689/ using document root <none> and password file <none>
I20260929 21:02:42.339969 14317 fs_manager.cc:359] Metadata directory not provided
I20260929 21:02:42.340098 14317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:02:42.340391 14317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:02:42.343245 14317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/data/instance:
uuid: "fa23bbba7a9042d8a58b0e95f853c442"
format_stamp: "Formatted at 2026-09-29 21:02:42 on dist-test-slave-1dms"
server_key: "435963ed4503eab4e00650c9268868d8"
server_key_iv: "5becfc92daa06d74e00ba998bc25fc17"
server_key_version: "encryptionkey@0"
I20260929 21:02:42.343991 14317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/wal/instance:
uuid: "fa23bbba7a9042d8a58b0e95f853c442"
format_stamp: "Formatted at 2026-09-29 21:02:42 on dist-test-slave-1dms"
server_key: "435963ed4503eab4e00650c9268868d8"
server_key_iv: "5becfc92daa06d74e00ba998bc25fc17"
server_key_version: "encryptionkey@0"
I20260929 21:02:42.348765 14317 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260929 21:02:42.352015 14332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:42.353353 14317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:02:42.353510 14317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/wal
uuid: "fa23bbba7a9042d8a58b0e95f853c442"
format_stamp: "Formatted at 2026-09-29 21:02:42 on dist-test-slave-1dms"
server_key: "435963ed4503eab4e00650c9268868d8"
server_key_iv: "5becfc92daa06d74e00ba998bc25fc17"
server_key_version: "encryptionkey@0"
I20260929 21:02:42.353677 14317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:02:42.380213 14317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:02:42.384040 14317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:02:42.384318 14317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:02:42.385275 14317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:02:42.386530 14317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:02:42.386601 14317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:42.386760 14317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:02:42.386821 14317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:42.418434 14317 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38137
I20260929 21:02:42.418503 14444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38137 every 8 connection(s)
I20260929 21:02:42.419652 14317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
I20260929 21:02:42.428519 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 14317
I20260929 21:02:42.428715 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/wal/instance
I20260929 21:02:42.429396 388 external_mini_cluster.cc:1428] Setting key 697349c76f29c09eca2c7ae30ca242f2
Sep 29 21:02:42 dist-test-slave-1dms krb5kdc[14211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:02:42.439666 14447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:42.422127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36497 (local address 127.0.97.62:39983)
0929 21:02:42.422588 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:42.422594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:42.423688 (+ 1094us) server_negotiation.cc:407] Connection header received
0929 21:02:42.425529 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:42.425534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:42.425598 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:42.425683 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:42.427127 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:42.427828 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:42.429570 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:42.430046 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:42.430191 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:42.434101 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:42.434106 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:42.434109 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:42.434140 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:42.436676 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:42.437434 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:42.437438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:42.437440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:42.437559 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:42.437975 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:42.437980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:42.437983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:42.438235 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:42.438391 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:42.439076 (+ 685us) server_negotiation.cc:299] Negotiation successful
0929 21:02:42.439308 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":221,"threads_started":1}
I20260929 21:02:42.440621 14448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:42.422758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.1:36497)
0929 21:02:42.423534 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:42.423571 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 21:02:42.425233 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:02:42.425884 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:02:42.425892 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:02:42.426418 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:02:42.426977 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:42.426995 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:42.427954 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:42.427957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:42.429389 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:42.429406 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:42.430305 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:42.430312 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:42.430555 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:42.431261 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:02:42.431299 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:02:42.433894 (+ 2595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:02:42.436817 (+ 2923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:42.436827 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:42.436870 (+ 43us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:42.437303 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:42.437666 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:42.437672 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:42.437675 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:42.437861 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:42.438362 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:02:42.438370 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:02:42.438706 (+ 336us) client_negotiation.cc:769] Sending connection context
0929 21:02:42.438963 (+ 257us) client_negotiation.cc:240] Negotiation successful
0929 21:02:42.439491 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":287,"threads_started":1}
I20260929 21:02:42.442126 14445 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39983
I20260929 21:02:42.442420 14445 heartbeater.cc:464] Registering TS with master...
I20260929 21:02:42.443089 14445 heartbeater.cc:511] Master 127.0.97.62:39983 requested a full tablet report, sending...
I20260929 21:02:42.445274 14256 ts_manager.cc:194] Registered new tserver with Master: fa23bbba7a9042d8a58b0e95f853c442 (127.0.97.1:38137)
I20260929 21:02:42.447474 14256 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:36497
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 21:02:42.480299 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39983
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:02:42.806762 14451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:02:42.807024 14451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:02:42.807199 14451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:02:42.814481 14451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:02:42.814591 14451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:02:42.814776 14451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 21:02:42.824440 14451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:02:42.825757 14451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:02:42.827061 14451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:02:42.840905 14457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:42.841187 14459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:42.841023 14456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:02:42.842097 14451 server_base.cc:1034] running on GCE node
I20260929 21:02:42.842694 14451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:02:42.843637 14451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:02:42.844918 14451 hybrid_clock.cc:648] HybridClock initialized: now 1790715762844848 us; error 51 us; skew 500 ppm
I20260929 21:02:42.845324 14451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:02:42 dist-test-slave-1dms krb5kdc[14211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:02:42.848389 14451 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 21:02:42.849808 14451 webserver.cc:460] Webserver started at http://127.0.97.2:43629/ using document root <none> and password file <none>
I20260929 21:02:42.850440 14451 fs_manager.cc:359] Metadata directory not provided
I20260929 21:02:42.850536 14451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:02:42.850773 14451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:02:42.853724 14451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/data/instance:
uuid: "059743faeb2246d5b6a5510034eaa7cb"
format_stamp: "Formatted at 2026-09-29 21:02:42 on dist-test-slave-1dms"
server_key: "35491f0d67ca90a535ce21725f820bdc"
server_key_iv: "7818b9b569bcdb1f151e9d699149c3a9"
server_key_version: "encryptionkey@0"
I20260929 21:02:42.854518 14451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/wal/instance:
uuid: "059743faeb2246d5b6a5510034eaa7cb"
format_stamp: "Formatted at 2026-09-29 21:02:42 on dist-test-slave-1dms"
server_key: "35491f0d67ca90a535ce21725f820bdc"
server_key_iv: "7818b9b569bcdb1f151e9d699149c3a9"
server_key_version: "encryptionkey@0"
I20260929 21:02:42.859687 14451 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 21:02:42.863077 14466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:42.864302 14451 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 21:02:42.864463 14451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/wal
uuid: "059743faeb2246d5b6a5510034eaa7cb"
format_stamp: "Formatted at 2026-09-29 21:02:42 on dist-test-slave-1dms"
server_key: "35491f0d67ca90a535ce21725f820bdc"
server_key_iv: "7818b9b569bcdb1f151e9d699149c3a9"
server_key_version: "encryptionkey@0"
I20260929 21:02:42.864650 14451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:02:42.888669 14451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:02:42.893187 14451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:02:42.893468 14451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:02:42.894461 14451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:02:42.895860 14451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:02:42.895933 14451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:42.896086 14451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:02:42.896144 14451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:42.929317 14451 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45349
I20260929 21:02:42.929399 14578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45349 every 8 connection(s)
I20260929 21:02:42.930613 14451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
I20260929 21:02:42.931246 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 14451
I20260929 21:02:42.931437 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/wal/instance
I20260929 21:02:42.932020 388 external_mini_cluster.cc:1428] Setting key 1f6335274de0ba8f1fe40b5875a821f6
Sep 29 21:02:42 dist-test-slave-1dms krb5kdc[14211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:02:42.950258 14447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:42.933388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49463 (local address 127.0.97.62:39983)
0929 21:02:42.933560 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:42.933564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:42.935039 (+ 1475us) server_negotiation.cc:407] Connection header received
0929 21:02:42.936880 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:42.936885 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:42.936975 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:42.937080 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:42.938539 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:42.939227 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:42.940811 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:42.941259 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:42.941392 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:42.944668 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:42.944673 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:42.944676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:42.944707 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:42.947246 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:42.947951 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:42.947958 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:42.947961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:42.948110 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:42.948525 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:42.948531 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:42.948535 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:42.948790 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:42.949014 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:42.949666 (+ 652us) server_negotiation.cc:299] Negotiation successful
0929 21:02:42.949896 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 21:02:42.951203 14581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:42.934034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.2:49463)
0929 21:02:42.934877 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:42.934921 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 21:02:42.936580 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:02:42.937310 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:02:42.937318 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:02:42.937828 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:02:42.938370 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:42.938386 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:42.939383 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:42.939393 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:42.940675 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:42.940686 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:42.941488 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:42.941495 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:42.941716 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:42.942375 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:02:42.942396 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:02:42.944463 (+ 2067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:02:42.947400 (+ 2937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:42.947408 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:42.947423 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:42.947836 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:42.948225 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:42.948230 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:42.948233 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:42.948417 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:42.948992 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:02:42.949001 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:02:42.949318 (+ 317us) client_negotiation.cc:769] Sending connection context
0929 21:02:42.949576 (+ 258us) client_negotiation.cc:240] Negotiation successful
0929 21:02:42.950097 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":418,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 21:02:42.952653 14579 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39983
I20260929 21:02:42.953002 14579 heartbeater.cc:464] Registering TS with master...
I20260929 21:02:42.953801 14579 heartbeater.cc:511] Master 127.0.97.62:39983 requested a full tablet report, sending...
I20260929 21:02:42.955369 14256 ts_manager.cc:194] Registered new tserver with Master: 059743faeb2246d5b6a5510034eaa7cb (127.0.97.2:45349)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 21:02:42.956481 14256 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:49463
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 21:02:42.983554 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39983
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:02:43.299288 14584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:02:43.299575 14584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:02:43.299727 14584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:02:43.307041 14584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:02:43.307150 14584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:02:43.307355 14584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 21:02:43.317014 14584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:02:43.318290 14584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:02:43.319572 14584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:02:43.332321 14589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:43.332285 14590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:43.332245 14592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:02:43.333155 14584 server_base.cc:1034] running on GCE node
I20260929 21:02:43.333797 14584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:02:43.334684 14584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:02:43.335882 14584 hybrid_clock.cc:648] HybridClock initialized: now 1790715763335817 us; error 63 us; skew 500 ppm
I20260929 21:02:43.336169 14584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:02:43 dist-test-slave-1dms krb5kdc[14211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:02:43.339241 14584 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 21:02:43.340744 14584 webserver.cc:460] Webserver started at http://127.0.97.3:39693/ using document root <none> and password file <none>
I20260929 21:02:43.341428 14584 fs_manager.cc:359] Metadata directory not provided
I20260929 21:02:43.341564 14584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:02:43.341851 14584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:02:43.344645 14584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/data/instance:
uuid: "c2dd5933f6464fa8a3ce2af782eacfb4"
format_stamp: "Formatted at 2026-09-29 21:02:43 on dist-test-slave-1dms"
server_key: "4b1b0d413af3cabbe47560980d78c6ad"
server_key_iv: "11ef40d280e0524898351c160bd39235"
server_key_version: "encryptionkey@0"
I20260929 21:02:43.345441 14584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c2dd5933f6464fa8a3ce2af782eacfb4"
format_stamp: "Formatted at 2026-09-29 21:02:43 on dist-test-slave-1dms"
server_key: "4b1b0d413af3cabbe47560980d78c6ad"
server_key_iv: "11ef40d280e0524898351c160bd39235"
server_key_version: "encryptionkey@0"
I20260929 21:02:43.350173 14584 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 21:02:43.353308 14599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:43.354552 14584 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 21:02:43.354717 14584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/wal
uuid: "c2dd5933f6464fa8a3ce2af782eacfb4"
format_stamp: "Formatted at 2026-09-29 21:02:43 on dist-test-slave-1dms"
server_key: "4b1b0d413af3cabbe47560980d78c6ad"
server_key_iv: "11ef40d280e0524898351c160bd39235"
server_key_version: "encryptionkey@0"
I20260929 21:02:43.354903 14584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:02:43.380203 14584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:02:43.384012 14584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:02:43.384313 14584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:02:43.385313 14584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:02:43.386590 14584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:02:43.386688 14584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:43.386828 14584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:02:43.386896 14584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:43.417578 14584 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40965
I20260929 21:02:43.417623 14711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40965 every 8 connection(s)
I20260929 21:02:43.419186 14584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
I20260929 21:02:43.425557 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 14584
I20260929 21:02:43.425745 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/wal/instance
I20260929 21:02:43.426175 388 external_mini_cluster.cc:1428] Setting key 6131276b10d9e091ce5f4ab22752ec87
Sep 29 21:02:43 dist-test-slave-1dms krb5kdc[14211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:02:43.440192 14447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:43.422073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51815 (local address 127.0.97.62:39983)
0929 21:02:43.422285 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:43.422291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:43.424175 (+ 1884us) server_negotiation.cc:407] Connection header received
0929 21:02:43.426187 (+ 2012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:43.426192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:43.426314 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:43.426430 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:43.428070 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:43.429203 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:43.430744 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:43.431129 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:43.431273 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:43.434699 (+ 3426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:43.434704 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:43.434707 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:43.434741 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:43.437236 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:43.437948 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:43.437958 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:43.437960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:43.438086 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:43.438496 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:43.438502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:43.438505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:43.438770 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:43.438929 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:43.439601 (+ 672us) server_negotiation.cc:299] Negotiation successful
0929 21:02:43.439816 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 21:02:43.441370 14714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:43.422496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.3:51815)
0929 21:02:43.424012 (+ 1516us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:43.424049 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 21:02:43.425891 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:02:43.426674 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:02:43.426683 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:02:43.427334 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:02:43.427909 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:43.427927 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:43.429359 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:43.429364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:43.430588 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:43.430600 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:43.431358 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:43.431368 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:43.431605 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:43.432241 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:02:43.432273 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:02:43.434474 (+ 2201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:02:43.437375 (+ 2901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:43.437384 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:43.437400 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:43.437821 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:43.438193 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:43.438199 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:43.438202 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:43.438384 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:43.438908 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:02:43.438916 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:02:43.439237 (+ 321us) client_negotiation.cc:769] Sending connection context
0929 21:02:43.439504 (+ 267us) client_negotiation.cc:240] Negotiation successful
0929 21:02:43.440043 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1294,"thread_start_us":1089,"threads_started":1}
I20260929 21:02:43.442672 14712 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39983
I20260929 21:02:43.442979 14712 heartbeater.cc:464] Registering TS with master...
I20260929 21:02:43.443653 14712 heartbeater.cc:511] Master 127.0.97.62:39983 requested a full tablet report, sending...
I20260929 21:02:43.445039 14256 ts_manager.cc:194] Registered new tserver with Master: c2dd5933f6464fa8a3ce2af782eacfb4 (127.0.97.3:40965)
I20260929 21:02:43.445830 14256 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51815
I20260929 21:02:43.450320 14445 heartbeater.cc:503] Master 127.0.97.62:39983 was elected leader, sending a full tablet report...
I20260929 21:02:43.453894 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 21:02:43.470528 14447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:43.458484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45392 (local address 127.0.97.62:39983)
0929 21:02:43.458641 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:43.458645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:43.459114 (+ 469us) server_negotiation.cc:407] Connection header received
0929 21:02:43.459241 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:43.459244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:43.459306 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:43.459386 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:43.460684 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:43.461422 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:43.462714 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:43.463093 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:43.463183 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:43.465611 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:43.465616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:43.465619 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:43.465648 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:43.468079 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:43.468740 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:43.468744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:43.468746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:43.468920 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:43.469303 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:43.469310 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:43.469315 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:43.469615 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:43.469816 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:43.470051 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 21:02:43.470267 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 21:02:43.474539 14256 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 21:02:43.481413 14256 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 21:02:43.506572 14724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:43.493445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38137 (local address 127.0.0.1:39766)
0929 21:02:43.495619 (+ 2174us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:43.495642 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 21:02:43.495788 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:02:43.498541 (+ 2753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:02:43.498548 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:02:43.498576 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:02:43.499003 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:43.499016 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:43.501058 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:43.501063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:43.503659 (+ 2596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:43.503673 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:43.505574 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:43.505579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:43.505766 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:43.505822 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 21:02:43.505955 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 21:02:43.506147 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1927,"mutex_wait_us":1,"thread_start_us":329,"threads_started":1}
I20260929 21:02:43.506598 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:43.493747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39766 (local address 127.0.97.1:38137)
0929 21:02:43.497921 (+ 4174us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:43.497929 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:43.497956 (+ 27us) server_negotiation.cc:407] Connection header received
0929 21:02:43.498050 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:43.498059 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:43.498324 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:43.499243 (+ 919us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:02:43.499458 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:43.500909 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:43.503848 (+ 2939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:43.505376 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:43.505496 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:43.505736 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:43.506014 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 21:02:43.506250 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3997,"thread_start_us":3849,"threads_started":1}
I20260929 21:02:43.507916 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:43.493847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57432 (local address 127.0.97.3:40965)
0929 21:02:43.496458 (+ 2611us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:43.496467 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:43.496503 (+ 36us) server_negotiation.cc:407] Connection header received
0929 21:02:43.496611 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:43.496621 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:43.496922 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:43.497139 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:02:43.498617 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:43.501415 (+ 2798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:43.504919 (+ 3504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:43.506292 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:43.506419 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:43.506644 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:43.507421 (+ 777us) server_negotiation.cc:299] Negotiation successful
0929 21:02:43.507578 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2428,"thread_start_us":2243,"threads_started":1}
I20260929 21:02:43.507977 14725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:43.493316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40965 (local address 127.0.0.1:57432)
0929 21:02:43.494195 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:43.494242 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 21:02:43.494432 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:02:43.497928 (+ 3496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:02:43.497937 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:02:43.497969 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:02:43.498409 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:43.498426 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:43.501572 (+ 3146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:43.501577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:43.504722 (+ 3145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:43.504737 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:43.507032 (+ 2295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:43.507041 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:43.507234 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:43.507261 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 21:02:43.507383 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 21:02:43.507518 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"mutex_wait_us":662,"thread_start_us":1881,"threads_started":1}
I20260929 21:02:43.511063 14380 tablet_service.cc:1467] Processing CreateTablet for tablet 189c283c6cf440209f4942adfe8a36ab (DEFAULT_TABLE table=test-table [id=3f36276370814a80abd715e3996eb799]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:02:43.511462 14647 tablet_service.cc:1467] Processing CreateTablet for tablet 189c283c6cf440209f4942adfe8a36ab (DEFAULT_TABLE table=test-table [id=3f36276370814a80abd715e3996eb799]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:02:43.512475 14728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:43.495555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45349 (local address 127.0.0.1:35332)
0929 21:02:43.502610 (+ 7055us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:43.502637 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 21:02:43.502741 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:02:43.504966 (+ 2225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:02:43.504970 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:02:43.504989 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:02:43.505238 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:43.505244 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:43.507148 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:43.507153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:43.509862 (+ 2709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:43.509876 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:43.511719 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:43.511725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:43.511886 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:43.511926 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 21:02:43.512023 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 21:02:43.512152 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6915,"thread_start_us":6757,"threads_started":1}
I20260929 21:02:43.512624 14729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:43.495852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35332 (local address 127.0.97.2:45349)
0929 21:02:43.504104 (+ 8252us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:43.504111 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:43.504133 (+ 22us) server_negotiation.cc:407] Connection header received
0929 21:02:43.504242 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:43.504250 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:43.504483 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:43.504674 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:02:43.505391 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:43.506963 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:43.510070 (+ 3107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:43.511501 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:43.511638 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:43.511835 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:43.512106 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 21:02:43.512317 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8108,"thread_start_us":7990,"threads_started":1}
I20260929 21:02:43.512784 14380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 189c283c6cf440209f4942adfe8a36ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:02:43.512902 14647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 189c283c6cf440209f4942adfe8a36ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:02:43.516121 14514 tablet_service.cc:1467] Processing CreateTablet for tablet 189c283c6cf440209f4942adfe8a36ab (DEFAULT_TABLE table=test-table [id=3f36276370814a80abd715e3996eb799]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:02:43.517522 14514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 189c283c6cf440209f4942adfe8a36ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:02:43.524152 14730 tablet_bootstrap.cc:492] T 189c283c6cf440209f4942adfe8a36ab P fa23bbba7a9042d8a58b0e95f853c442: Bootstrap starting.
I20260929 21:02:43.527477 14730 tablet_bootstrap.cc:654] T 189c283c6cf440209f4942adfe8a36ab P fa23bbba7a9042d8a58b0e95f853c442: Neither blocks nor log segments found. Creating new log.
I20260929 21:02:43.528437 14730 log.cc:824] T 189c283c6cf440209f4942adfe8a36ab P fa23bbba7a9042d8a58b0e95f853c442: Log is configured to *not* fsync() on all Append() calls
I20260929 21:02:43.529687 14731 tablet_bootstrap.cc:492] T 189c283c6cf440209f4942adfe8a36ab P c2dd5933f6464fa8a3ce2af782eacfb4: Bootstrap starting.
I20260929 21:02:43.532379 14733 tablet_bootstrap.cc:492] T 189c283c6cf440209f4942adfe8a36ab P 059743faeb2246d5b6a5510034eaa7cb: Bootstrap starting.
I20260929 21:02:43.533301 14731 tablet_bootstrap.cc:654] T 189c283c6cf440209f4942adfe8a36ab P c2dd5933f6464fa8a3ce2af782eacfb4: Neither blocks nor log segments found. Creating new log.
I20260929 21:02:43.533687 14730 tablet_bootstrap.cc:492] T 189c283c6cf440209f4942adfe8a36ab P fa23bbba7a9042d8a58b0e95f853c442: No bootstrap required, opened a new log
I20260929 21:02:43.533979 14730 ts_tablet_manager.cc:1397] T 189c283c6cf440209f4942adfe8a36ab P fa23bbba7a9042d8a58b0e95f853c442: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260929 21:02:43.534507 14731 log.cc:824] T 189c283c6cf440209f4942adfe8a36ab P c2dd5933f6464fa8a3ce2af782eacfb4: Log is configured to *not* fsync() on all Append() calls
I20260929 21:02:43.535431 14733 tablet_bootstrap.cc:654] T 189c283c6cf440209f4942adfe8a36ab P 059743faeb2246d5b6a5510034eaa7cb: Neither blocks nor log segments found. Creating new log.
I20260929 21:02:43.536446 14733 log.cc:824] T 189c283c6cf440209f4942adfe8a36ab P 059743faeb2246d5b6a5510034eaa7cb: Log is configured to *not* fsync() on all Append() calls
I20260929 21:02:43.537541 14731 tablet_bootstrap.cc:492] T 189c283c6cf440209f4942adfe8a36ab P c2dd5933f6464fa8a3ce2af782eacfb4: No bootstrap required, opened a new log
I20260929 21:02:43.537849 14731 ts_tablet_manager.cc:1397] T 189c283c6cf440209f4942adfe8a36ab P c2dd5933f6464fa8a3ce2af782eacfb4: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260929 21:02:43.538812 14733 tablet_bootstrap.cc:492] T 189c283c6cf440209f4942adfe8a36ab P 059743faeb2246d5b6a5510034eaa7cb: No bootstrap required, opened a new log
I20260929 21:02:43.539053 14733 ts_tablet_manager.cc:1397] T 189c283c6cf440209f4942adfe8a36ab P 059743faeb2246d5b6a5510034eaa7cb: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260929 21:02:43.544623 14730 raft_consensus.cc:348] T 189c283c6cf440209f4942adfe8a36ab P fa23bbba7a9042d8a58b0e95f853c442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dd5933f6464fa8a3ce2af782eacfb4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40965 } } peers { permanent_uuid: "fa23bbba7a9042d8a58b0e95f853c442" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38137 } } peers { permanent_uuid: "059743faeb2246d5b6a5510034eaa7cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45349 } }
I20260929 21:02:43.545042 14730 raft_consensus.cc:374] T 189c283c6cf440209f4942adfe8a36ab P fa23bbba7a9042d8a58b0e95f853c442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:02:43.545166 14730 raft_consensus.cc:729] T 189c283c6cf440209f4942adfe8a36ab P fa23bbba7a9042d8a58b0e95f853c442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa23bbba7a9042d8a58b0e95f853c442, State: Initialized, Role: FOLLOWER
I20260929 21:02:43.545759 14730 consensus_queue.cc:260] T 189c283c6cf440209f4942adfe8a36ab P fa23bbba7a9042d8a58b0e95f853c442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dd5933f6464fa8a3ce2af782eacfb4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40965 } } peers { permanent_uuid: "fa23bbba7a9042d8a58b0e95f853c442" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38137 } } peers { permanent_uuid: "059743faeb2246d5b6a5510034eaa7cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45349 } }
I20260929 21:02:43.547448 14730 ts_tablet_manager.cc:1428] T 189c283c6cf440209f4942adfe8a36ab P fa23bbba7a9042d8a58b0e95f853c442: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260929 21:02:43.547715 14733 raft_consensus.cc:348] T 189c283c6cf440209f4942adfe8a36ab P 059743faeb2246d5b6a5510034eaa7cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dd5933f6464fa8a3ce2af782eacfb4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40965 } } peers { permanent_uuid: "fa23bbba7a9042d8a58b0e95f853c442" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38137 } } peers { permanent_uuid: "059743faeb2246d5b6a5510034eaa7cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45349 } }
I20260929 21:02:43.548098 14733 raft_consensus.cc:374] T 189c283c6cf440209f4942adfe8a36ab P 059743faeb2246d5b6a5510034eaa7cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:02:43.548173 14733 raft_consensus.cc:729] T 189c283c6cf440209f4942adfe8a36ab P 059743faeb2246d5b6a5510034eaa7cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059743faeb2246d5b6a5510034eaa7cb, State: Initialized, Role: FOLLOWER
I20260929 21:02:43.548012 14731 raft_consensus.cc:348] T 189c283c6cf440209f4942adfe8a36ab P c2dd5933f6464fa8a3ce2af782eacfb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dd5933f6464fa8a3ce2af782eacfb4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40965 } } peers { permanent_uuid: "fa23bbba7a9042d8a58b0e95f853c442" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38137 } } peers { permanent_uuid: "059743faeb2246d5b6a5510034eaa7cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45349 } }
I20260929 21:02:43.548388 14731 raft_consensus.cc:374] T 189c283c6cf440209f4942adfe8a36ab P c2dd5933f6464fa8a3ce2af782eacfb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:02:43.548471 14731 raft_consensus.cc:729] T 189c283c6cf440209f4942adfe8a36ab P c2dd5933f6464fa8a3ce2af782eacfb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2dd5933f6464fa8a3ce2af782eacfb4, State: Initialized, Role: FOLLOWER
I20260929 21:02:43.548629 14733 consensus_queue.cc:260] T 189c283c6cf440209f4942adfe8a36ab P 059743faeb2246d5b6a5510034eaa7cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dd5933f6464fa8a3ce2af782eacfb4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40965 } } peers { permanent_uuid: "fa23bbba7a9042d8a58b0e95f853c442" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38137 } } peers { permanent_uuid: "059743faeb2246d5b6a5510034eaa7cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45349 } }
I20260929 21:02:43.549005 14731 consensus_queue.cc:260] T 189c283c6cf440209f4942adfe8a36ab P c2dd5933f6464fa8a3ce2af782eacfb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dd5933f6464fa8a3ce2af782eacfb4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40965 } } peers { permanent_uuid: "fa23bbba7a9042d8a58b0e95f853c442" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38137 } } peers { permanent_uuid: "059743faeb2246d5b6a5510034eaa7cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45349 } }
I20260929 21:02:43.550117 14712 heartbeater.cc:503] Master 127.0.97.62:39983 was elected leader, sending a full tablet report...
I20260929 21:02:43.550877 14579 heartbeater.cc:503] Master 127.0.97.62:39983 was elected leader, sending a full tablet report...
I20260929 21:02:43.551591 14733 ts_tablet_manager.cc:1428] T 189c283c6cf440209f4942adfe8a36ab P 059743faeb2246d5b6a5510034eaa7cb: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
I20260929 21:02:43.552772 14731 ts_tablet_manager.cc:1428] T 189c283c6cf440209f4942adfe8a36ab P c2dd5933f6464fa8a3ce2af782eacfb4: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
I20260929 21:02:43.652171 14737 raft_consensus.cc:482] T 189c283c6cf440209f4942adfe8a36ab P 059743faeb2246d5b6a5510034eaa7cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 21:02:43.652432 14737 raft_consensus.cc:504] T 189c283c6cf440209f4942adfe8a36ab P 059743faeb2246d5b6a5510034eaa7cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dd5933f6464fa8a3ce2af782eacfb4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40965 } } peers { permanent_uuid: "fa23bbba7a9042d8a58b0e95f853c442" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38137 } } peers { permanent_uuid: "059743faeb2246d5b6a5510034eaa7cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45349 } }
I20260929 21:02:43.654243 14737 leader_election.cc:288] T 189c283c6cf440209f4942adfe8a36ab P 059743faeb2246d5b6a5510034eaa7cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2dd5933f6464fa8a3ce2af782eacfb4 (127.0.97.3:40965), fa23bbba7a9042d8a58b0e95f853c442 (127.0.97.1:38137)
I20260929 21:02:43.661307 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:43.654539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36171 (local address 127.0.97.3:40965)
0929 21:02:43.654769 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:43.654777 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:43.655586 (+ 809us) server_negotiation.cc:407] Connection header received
0929 21:02:43.655760 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:43.655767 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:43.655884 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:43.655961 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:02:43.656687 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:43.657489 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:43.659560 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:43.660335 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:43.660485 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:43.660601 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:43.660913 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 21:02:43.661035 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 21:02:43.661335 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:43.655039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53895 (local address 127.0.97.1:38137)
0929 21:02:43.655275 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:43.655281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:43.655757 (+ 476us) server_negotiation.cc:407] Connection header received
0929 21:02:43.655894 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:43.655898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:43.655984 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:43.656082 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:02:43.656741 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:43.657569 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:43.659521 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:43.660347 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:43.660494 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:43.660605 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:43.660955 (+ 350us) server_negotiation.cc:299] Negotiation successful
0929 21:02:43.661077 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 21:02:43.661520 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:43.654463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40965 (local address 127.0.97.2:36171)
0929 21:02:43.655429 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:43.655458 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 21:02:43.655628 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:02:43.656001 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:02:43.656006 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:02:43.656035 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:02:43.656527 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:43.656541 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:43.657613 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:43.657621 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:43.659383 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:43.659395 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:43.660565 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:43.660572 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:43.660729 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:43.660752 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 21:02:43.660882 (+ 130us) client_negotiation.cc:240] Negotiation successful
0929 21:02:43.661010 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"mutex_wait_us":31,"thread_start_us":380,"threads_started":1}
I20260929 21:02:43.661563 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:43.654643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38137 (local address 127.0.97.2:53895)
0929 21:02:43.655628 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:43.655643 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 21:02:43.655785 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:02:43.656122 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:02:43.656130 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:02:43.656167 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:02:43.656586 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:43.656610 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:43.657695 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:43.657706 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:43.659375 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:43.659389 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:43.660604 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:43.660612 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:43.660765 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:43.660789 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 21:02:43.660928 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 21:02:43.661051 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"mutex_wait_us":5,"spinlock_wait_cycles":33536,"thread_start_us":619,"threads_started":1}
I20260929 21:02:43.662487 14667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "189c283c6cf440209f4942adfe8a36ab" candidate_uuid: "059743faeb2246d5b6a5510034eaa7cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2dd5933f6464fa8a3ce2af782eacfb4" is_pre_election: true
I20260929 21:02:43.662480 14400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "189c283c6cf440209f4942adfe8a36ab" candidate_uuid: "059743faeb2246d5b6a5510034eaa7cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa23bbba7a9042d8a58b0e95f853c442" is_pre_election: true
I20260929 21:02:43.662978 14667 raft_consensus.cc:2454] T 189c283c6cf440209f4942adfe8a36ab P c2dd5933f6464fa8a3ce2af782eacfb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059743faeb2246d5b6a5510034eaa7cb in term 0.
I20260929 21:02:43.662978 14400 raft_consensus.cc:2454] T 189c283c6cf440209f4942adfe8a36ab P fa23bbba7a9042d8a58b0e95f853c442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059743faeb2246d5b6a5510034eaa7cb in term 0.
I20260929 21:02:43.663656 14467 leader_election.cc:302] T 189c283c6cf440209f4942adfe8a36ab P 059743faeb2246d5b6a5510034eaa7cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 059743faeb2246d5b6a5510034eaa7cb, c2dd5933f6464fa8a3ce2af782eacfb4, fa23bbba7a9042d8a58b0e95f853c442; no voters:
I20260929 21:02:43.664083 14737 raft_consensus.cc:2781] T 189c283c6cf440209f4942adfe8a36ab P 059743faeb2246d5b6a5510034eaa7cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 21:02:43.664233 14737 raft_consensus.cc:482] T 189c283c6cf440209f4942adfe8a36ab P 059743faeb2246d5b6a5510034eaa7cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 21:02:43.664346 14737 raft_consensus.cc:3037] T 189c283c6cf440209f4942adfe8a36ab P 059743faeb2246d5b6a5510034eaa7cb [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:02:43.666560 14737 raft_consensus.cc:504] T 189c283c6cf440209f4942adfe8a36ab P 059743faeb2246d5b6a5510034eaa7cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dd5933f6464fa8a3ce2af782eacfb4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40965 } } peers { permanent_uuid: "fa23bbba7a9042d8a58b0e95f853c442" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38137 } } peers { permanent_uuid: "059743faeb2246d5b6a5510034eaa7cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45349 } }
I20260929 21:02:43.667128 14737 leader_election.cc:288] T 189c283c6cf440209f4942adfe8a36ab P 059743faeb2246d5b6a5510034eaa7cb [CANDIDATE]: Term 1 election: Requested vote from peers c2dd5933f6464fa8a3ce2af782eacfb4 (127.0.97.3:40965), fa23bbba7a9042d8a58b0e95f853c442 (127.0.97.1:38137)
I20260929 21:02:43.667483 14667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "189c283c6cf440209f4942adfe8a36ab" candidate_uuid: "059743faeb2246d5b6a5510034eaa7cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2dd5933f6464fa8a3ce2af782eacfb4"
I20260929 21:02:43.667593 14400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "189c283c6cf440209f4942adfe8a36ab" candidate_uuid: "059743faeb2246d5b6a5510034eaa7cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa23bbba7a9042d8a58b0e95f853c442"
I20260929 21:02:43.667673 14667 raft_consensus.cc:3037] T 189c283c6cf440209f4942adfe8a36ab P c2dd5933f6464fa8a3ce2af782eacfb4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:02:43.667810 14400 raft_consensus.cc:3037] T 189c283c6cf440209f4942adfe8a36ab P fa23bbba7a9042d8a58b0e95f853c442 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:02:43.670156 14667 raft_consensus.cc:2454] T 189c283c6cf440209f4942adfe8a36ab P c2dd5933f6464fa8a3ce2af782eacfb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059743faeb2246d5b6a5510034eaa7cb in term 1.
I20260929 21:02:43.670156 14400 raft_consensus.cc:2454] T 189c283c6cf440209f4942adfe8a36ab P fa23bbba7a9042d8a58b0e95f853c442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059743faeb2246d5b6a5510034eaa7cb in term 1.
I20260929 21:02:43.670547 14469 leader_election.cc:302] T 189c283c6cf440209f4942adfe8a36ab P 059743faeb2246d5b6a5510034eaa7cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059743faeb2246d5b6a5510034eaa7cb, c2dd5933f6464fa8a3ce2af782eacfb4; no voters:
I20260929 21:02:43.670853 14737 raft_consensus.cc:2781] T 189c283c6cf440209f4942adfe8a36ab P 059743faeb2246d5b6a5510034eaa7cb [term 1 FOLLOWER]: Leader election won for term 1
W20260929 21:02:43.670919 14713 tablet.cc:2369] T 189c283c6cf440209f4942adfe8a36ab P c2dd5933f6464fa8a3ce2af782eacfb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 21:02:43.671367 14737 raft_consensus.cc:686] T 189c283c6cf440209f4942adfe8a36ab P 059743faeb2246d5b6a5510034eaa7cb [term 1 LEADER]: Becoming Leader. State: Replica: 059743faeb2246d5b6a5510034eaa7cb, State: Running, Role: LEADER
W20260929 21:02:43.671854 14446 tablet.cc:2369] T 189c283c6cf440209f4942adfe8a36ab P fa23bbba7a9042d8a58b0e95f853c442: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 21:02:43.671880 14737 consensus_queue.cc:237] T 189c283c6cf440209f4942adfe8a36ab P 059743faeb2246d5b6a5510034eaa7cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dd5933f6464fa8a3ce2af782eacfb4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40965 } } peers { permanent_uuid: "fa23bbba7a9042d8a58b0e95f853c442" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38137 } } peers { permanent_uuid: "059743faeb2246d5b6a5510034eaa7cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45349 } }
I20260929 21:02:43.676275 14255 catalog_manager.cc:5495] T 189c283c6cf440209f4942adfe8a36ab P 059743faeb2246d5b6a5510034eaa7cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 059743faeb2246d5b6a5510034eaa7cb (127.0.97.2). New cstate: current_term: 1 leader_uuid: "059743faeb2246d5b6a5510034eaa7cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dd5933f6464fa8a3ce2af782eacfb4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa23bbba7a9042d8a58b0e95f853c442" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059743faeb2246d5b6a5510034eaa7cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45349 } health_report { overall_health: HEALTHY } } }
W20260929 21:02:43.682670 14580 tablet.cc:2369] T 189c283c6cf440209f4942adfe8a36ab P 059743faeb2246d5b6a5510034eaa7cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 21:02:43.692790 14729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:43.687991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35346 (local address 127.0.97.2:45349)
0929 21:02:43.688160 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:43.688165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:43.688181 (+ 16us) server_negotiation.cc:407] Connection header received
0929 21:02:43.688306 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:43.688310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:43.688383 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:43.688502 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:02:43.688815 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:43.689556 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:43.690899 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:43.691275 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:43.691362 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:43.691802 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:02:43.692262 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:02:43.692375 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:43.692436 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 21:02:43.692497 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 21:02:43.700881 14667 raft_consensus.cc:1261] T 189c283c6cf440209f4942adfe8a36ab P c2dd5933f6464fa8a3ce2af782eacfb4 [term 1 FOLLOWER]: Refusing update from remote peer 059743faeb2246d5b6a5510034eaa7cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:02:43.700894 14400 raft_consensus.cc:1261] T 189c283c6cf440209f4942adfe8a36ab P fa23bbba7a9042d8a58b0e95f853c442 [term 1 FOLLOWER]: Refusing update from remote peer 059743faeb2246d5b6a5510034eaa7cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:02:43.701819 14741 consensus_queue.cc:1035] T 189c283c6cf440209f4942adfe8a36ab P 059743faeb2246d5b6a5510034eaa7cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa23bbba7a9042d8a58b0e95f853c442" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:02:43.702167 14737 consensus_queue.cc:1035] T 189c283c6cf440209f4942adfe8a36ab P 059743faeb2246d5b6a5510034eaa7cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2dd5933f6464fa8a3ce2af782eacfb4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:02:43.717712 14745 mvcc.cc:205] Tried to move back new op lower bound from 7334771768108888064 to 7334771768004108288. Current Snapshot: MvccSnapshot[applied={T|T < 7334771768108888064}]
I20260929 21:02:43.727833 14254 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 21:02:43.728106 14254 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 21:02:43.731695 14254 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16: Sending DeleteTablet for 3 replicas of tablet 189c283c6cf440209f4942adfe8a36ab
I20260929 21:02:43.732789 14380 tablet_service.cc:1514] Processing DeleteTablet for tablet 189c283c6cf440209f4942adfe8a36ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:02:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39766
I20260929 21:02:43.733085 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 14317
I20260929 21:02:43.733141 14647 tablet_service.cc:1514] Processing DeleteTablet for tablet 189c283c6cf440209f4942adfe8a36ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:02:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57432
I20260929 21:02:43.733798 14514 tablet_service.cc:1514] Processing DeleteTablet for tablet 189c283c6cf440209f4942adfe8a36ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:02:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35332
I20260929 21:02:43.745182 14755 tablet_replica.cc:330] T 189c283c6cf440209f4942adfe8a36ab P c2dd5933f6464fa8a3ce2af782eacfb4: stopping tablet replica
I20260929 21:02:43.745817 14755 raft_consensus.cc:2229] T 189c283c6cf440209f4942adfe8a36ab P c2dd5933f6464fa8a3ce2af782eacfb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:02:43.746407 14755 raft_consensus.cc:2258] T 189c283c6cf440209f4942adfe8a36ab P c2dd5933f6464fa8a3ce2af782eacfb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:02:43.748073 14757 tablet_replica.cc:330] T 189c283c6cf440209f4942adfe8a36ab P 059743faeb2246d5b6a5510034eaa7cb: stopping tablet replica
I20260929 21:02:43.748591 14757 raft_consensus.cc:2229] T 189c283c6cf440209f4942adfe8a36ab P 059743faeb2246d5b6a5510034eaa7cb [term 1 LEADER]: Raft consensus shutting down.
I20260929 21:02:43.750222 14757 raft_consensus.cc:2258] T 189c283c6cf440209f4942adfe8a36ab P 059743faeb2246d5b6a5510034eaa7cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:02:43.750433 14754 tablet_replica.cc:330] T 189c283c6cf440209f4942adfe8a36ab P fa23bbba7a9042d8a58b0e95f853c442: stopping tablet replica
I20260929 21:02:43.750811 14754 raft_consensus.cc:2229] T 189c283c6cf440209f4942adfe8a36ab P fa23bbba7a9042d8a58b0e95f853c442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:02:43.751313 14754 raft_consensus.cc:2258] T 189c283c6cf440209f4942adfe8a36ab P fa23bbba7a9042d8a58b0e95f853c442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:02:43.752542 14755 ts_tablet_manager.cc:1905] T 189c283c6cf440209f4942adfe8a36ab P c2dd5933f6464fa8a3ce2af782eacfb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:02:43.756042 14754 ts_tablet_manager.cc:1905] T 189c283c6cf440209f4942adfe8a36ab P fa23bbba7a9042d8a58b0e95f853c442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:02:43.758472 14757 ts_tablet_manager.cc:1905] T 189c283c6cf440209f4942adfe8a36ab P 059743faeb2246d5b6a5510034eaa7cb: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 21:02:43 dist-test-slave-1dms krb5kdc[14211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 21:02:43.763763 14754 ts_tablet_manager.cc:1918] T 189c283c6cf440209f4942adfe8a36ab P fa23bbba7a9042d8a58b0e95f853c442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:02:43.763933 14754 log.cc:1196] T 189c283c6cf440209f4942adfe8a36ab P fa23bbba7a9042d8a58b0e95f853c442: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/wal/wals/189c283c6cf440209f4942adfe8a36ab
I20260929 21:02:43.764452 14754 ts_tablet_manager.cc:1939] T 189c283c6cf440209f4942adfe8a36ab P fa23bbba7a9042d8a58b0e95f853c442: Deleting consensus metadata
I20260929 21:02:43.765961 14757 ts_tablet_manager.cc:1918] T 189c283c6cf440209f4942adfe8a36ab P 059743faeb2246d5b6a5510034eaa7cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:02:43.766134 14757 log.cc:1196] T 189c283c6cf440209f4942adfe8a36ab P 059743faeb2246d5b6a5510034eaa7cb: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/wal/wals/189c283c6cf440209f4942adfe8a36ab
I20260929 21:02:43.766490 14240 catalog_manager.cc:4841] TS fa23bbba7a9042d8a58b0e95f853c442 (127.0.97.1:38137): tablet 189c283c6cf440209f4942adfe8a36ab (table test-table [id=3f36276370814a80abd715e3996eb799]) successfully deleted
I20260929 21:02:43.766710 14757 ts_tablet_manager.cc:1939] T 189c283c6cf440209f4942adfe8a36ab P 059743faeb2246d5b6a5510034eaa7cb: Deleting consensus metadata
I20260929 21:02:43.768594 14242 catalog_manager.cc:4841] TS 059743faeb2246d5b6a5510034eaa7cb (127.0.97.2:45349): tablet 189c283c6cf440209f4942adfe8a36ab (table test-table [id=3f36276370814a80abd715e3996eb799]) successfully deleted
I20260929 21:02:43.773572 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:43.734461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39778 (local address 127.0.97.1:38137)
0929 21:02:43.734685 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:43.734695 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:43.747495 (+ 12800us) server_negotiation.cc:407] Connection header received
0929 21:02:43.749559 (+ 2064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:43.749569 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:43.749730 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:43.749867 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:43.753024 (+ 3157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:43.754392 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:43.757030 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:43.757891 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:43.758223 (+ 332us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:43.765277 (+ 7054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:43.765293 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:43.765311 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:43.765369 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:43.769342 (+ 3973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:43.770829 (+ 1487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:43.770839 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:43.770842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:43.771007 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:43.771473 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:43.771478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:43.771482 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:43.771918 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:43.772224 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:43.772803 (+ 579us) server_negotiation.cc:299] Negotiation successful
0929 21:02:43.773056 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 21:02:43.775185 14440 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:02:43.779165 14755 ts_tablet_manager.cc:1918] T 189c283c6cf440209f4942adfe8a36ab P c2dd5933f6464fa8a3ce2af782eacfb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:02:43.779379 14755 log.cc:1196] T 189c283c6cf440209f4942adfe8a36ab P c2dd5933f6464fa8a3ce2af782eacfb4: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/wal/wals/189c283c6cf440209f4942adfe8a36ab
I20260929 21:02:43.780253 14755 ts_tablet_manager.cc:1939] T 189c283c6cf440209f4942adfe8a36ab P c2dd5933f6464fa8a3ce2af782eacfb4: Deleting consensus metadata
I20260929 21:02:43.783224 14242 catalog_manager.cc:4841] TS c2dd5933f6464fa8a3ce2af782eacfb4 (127.0.97.3:40965): tablet 189c283c6cf440209f4942adfe8a36ab (table test-table [id=3f36276370814a80abd715e3996eb799]) successfully deleted
I20260929 21:02:43.884945 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 14317
I20260929 21:02:43.902869 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 14451
Sep 29 21:02:43 dist-test-slave-1dms krb5kdc[14211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 21:02:43.920416 14729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:43.903748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35358 (local address 127.0.97.2:45349)
0929 21:02:43.903910 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:43.903918 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:43.903935 (+ 17us) server_negotiation.cc:407] Connection header received
0929 21:02:43.904046 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:43.904049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:43.904120 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:43.904211 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:43.906004 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:43.906804 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:43.908651 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:43.909296 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:43.909393 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:43.914414 (+ 5021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:43.914428 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:43.914444 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:43.914483 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:43.917717 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:43.918372 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:43.918385 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:43.918388 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:43.918532 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:43.918895 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:43.918900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:43.918911 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:43.919263 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:43.919593 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:43.919838 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 21:02:43.920060 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 21:02:43.921213 14574 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:02:44.034457 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 14451
I20260929 21:02:44.053516 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 14584
Sep 29 21:02:44 dist-test-slave-1dms krb5kdc[14211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 21:02:44.068535 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:44.054185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57448 (local address 127.0.97.3:40965)
0929 21:02:44.054344 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:44.054349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:44.054410 (+ 61us) server_negotiation.cc:407] Connection header received
0929 21:02:44.054556 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:44.054560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:44.054626 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:44.054714 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:44.056036 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:44.056922 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:44.058358 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:44.058799 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:44.058895 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:44.063122 (+ 4227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:44.063137 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:44.063152 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:44.063195 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:44.066131 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:44.066835 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:44.066842 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:44.066844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:44.066983 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:44.067271 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:44.067274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:44.067276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:44.067594 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:44.067856 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:44.068084 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 21:02:44.068275 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 21:02:44.069204 14707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:02:44.158215 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 14584
I20260929 21:02:44.175033 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 14224
I20260929 21:02:44.176168 14286 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:02:44.286346 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 14224
I20260929 21:02:44.308350 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39983
--webserver_interface=127.0.97.62
--webserver_port=39115
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:02:44.634279 14762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:02:44.634599 14762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:02:44.634693 14762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:02:44.642061 14762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:02:44.642177 14762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:02:44.642223 14762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:02:44.642284 14762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:02:44.642328 14762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:02:44.652336 14762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=39115
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:02:44.653614 14762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:02:44.654844 14762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:02:44.677855 14769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:44.677874 14768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:44.677904 14771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:02:44.678768 14762 server_base.cc:1034] running on GCE node
I20260929 21:02:44.679679 14762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:02:44.681322 14762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:02:44.682551 14762 hybrid_clock.cc:648] HybridClock initialized: now 1790715764682498 us; error 40 us; skew 500 ppm
I20260929 21:02:44.682976 14762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:02:44 dist-test-slave-1dms krb5kdc[14211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:02:44.686507 14762 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 21:02:44.688313 14762 webserver.cc:460] Webserver started at http://127.0.97.62:39115/ using document root <none> and password file <none>
I20260929 21:02:44.689146 14762 fs_manager.cc:359] Metadata directory not provided
I20260929 21:02:44.689304 14762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:02:44.695220 14762 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 21:02:44.698130 14778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:44.699484 14762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:02:44.699677 14762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "bda88db6437c46fda958bcb092445d16"
format_stamp: "Formatted at 2026-09-29 21:02:41 on dist-test-slave-1dms"
server_key: "1eff2d0bff18c77852696b1f5d453896"
server_key_iv: "3593af8c53c54a3d5497dd17e8de5696"
server_key_version: "encryptionkey@0"
I20260929 21:02:44.700748 14762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:02:44.718530 14762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:02:44.722481 14762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:02:44.722766 14762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:02:44.741453 14762 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39983
I20260929 21:02:44.741463 14829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39983 every 8 connection(s)
I20260929 21:02:44.742933 14762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:02:44.749466 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 14762
I20260929 21:02:44.750005 14830 sys_catalog.cc:263] Verifying existing consensus state
I20260929 21:02:44.750486 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:38137
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:39983
--webserver_port=42689
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 21:02:44.755337 14830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16: Bootstrap starting.
I20260929 21:02:44.776240 14830 log.cc:824] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16: Log is configured to *not* fsync() on all Append() calls
I20260929 21:02:44.788317 14830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 21:02:44.788969 14830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16: Bootstrap complete.
I20260929 21:02:44.798110 14830 raft_consensus.cc:348] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda88db6437c46fda958bcb092445d16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39983 } }
I20260929 21:02:44.798988 14830 raft_consensus.cc:729] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bda88db6437c46fda958bcb092445d16, State: Initialized, Role: FOLLOWER
I20260929 21:02:44.799587 14830 consensus_queue.cc:260] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda88db6437c46fda958bcb092445d16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39983 } }
I20260929 21:02:44.799814 14830 raft_consensus.cc:388] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:02:44.799928 14830 raft_consensus.cc:482] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:02:44.800112 14830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [term 1 FOLLOWER]: Advancing to term 2
I20260929 21:02:44.803814 14830 raft_consensus.cc:504] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda88db6437c46fda958bcb092445d16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39983 } }
I20260929 21:02:44.804370 14830 leader_election.cc:302] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bda88db6437c46fda958bcb092445d16; no voters:
I20260929 21:02:44.805023 14830 leader_election.cc:288] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 21:02:44.805172 14834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 21:02:44.807570 14830 sys_catalog.cc:564] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:02:44.809113 14834 raft_consensus.cc:686] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [term 2 LEADER]: Becoming Leader. State: Replica: bda88db6437c46fda958bcb092445d16, State: Running, Role: LEADER
I20260929 21:02:44.809784 14834 consensus_queue.cc:237] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda88db6437c46fda958bcb092445d16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39983 } }
I20260929 21:02:44.812426 14835 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "bda88db6437c46fda958bcb092445d16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda88db6437c46fda958bcb092445d16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39983 } } }
I20260929 21:02:44.812790 14835 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [sys.catalog]: This master's current role is: LEADER
I20260929 21:02:44.813299 14836 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bda88db6437c46fda958bcb092445d16. Latest consensus state: current_term: 2 leader_uuid: "bda88db6437c46fda958bcb092445d16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda88db6437c46fda958bcb092445d16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39983 } } }
I20260929 21:02:44.813625 14836 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [sys.catalog]: This master's current role is: LEADER
I20260929 21:02:44.822079 14841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:02:44.832903 14841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:02:44.834692 14841 catalog_manager.cc:1259] Loaded cluster ID: 638bfd7a910c45d59fcd4bff57f4e24d
I20260929 21:02:44.834810 14841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:02:44.838735 14841 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 21:02:44.864046 14841 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 21:02:44.889791 14841 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:02:44.891805 14841 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16: Loaded TSK: 0
I20260929 21:02:44.893498 14841 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 21:02:45.112041 14832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:02:45.112323 14832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:02:45.112465 14832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:02:45.119807 14832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:02:45.119913 14832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:02:45.120075 14832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 21:02:45.130676 14832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:38137
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=42689
--tserver_master_addrs=127.0.97.62:39983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:02:45.132087 14832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:02:45.133525 14832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:02:45.149174 14858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:45.149173 14857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:45.150208 14860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:02:45.150208 14832 server_base.cc:1034] running on GCE node
I20260929 21:02:45.151062 14832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:02:45.152127 14832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:02:45.153357 14832 hybrid_clock.cc:648] HybridClock initialized: now 1790715765153290 us; error 73 us; skew 500 ppm
I20260929 21:02:45.153712 14832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:02:45 dist-test-slave-1dms krb5kdc[14211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:02:45.157157 14832 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 21:02:45.158852 14832 webserver.cc:460] Webserver started at http://127.0.97.1:42689/ using document root <none> and password file <none>
I20260929 21:02:45.159572 14832 fs_manager.cc:359] Metadata directory not provided
I20260929 21:02:45.159744 14832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:02:45.165746 14832 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 21:02:45.169339 14867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:45.171185 14832 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 21:02:45.171391 14832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/wal
uuid: "fa23bbba7a9042d8a58b0e95f853c442"
format_stamp: "Formatted at 2026-09-29 21:02:42 on dist-test-slave-1dms"
server_key: "435963ed4503eab4e00650c9268868d8"
server_key_iv: "5becfc92daa06d74e00ba998bc25fc17"
server_key_version: "encryptionkey@0"
I20260929 21:02:45.172876 14832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:02:45.219270 14832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:02:45.224660 14832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:02:45.224973 14832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:02:45.226202 14832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:02:45.227810 14832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:02:45.227927 14832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:45.228027 14832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:02:45.228101 14832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:45.261953 14832 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38137
I20260929 21:02:45.262045 14979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38137 every 8 connection(s)
I20260929 21:02:45.263409 14832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
I20260929 21:02:45.267349 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 14832
I20260929 21:02:45.267956 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:45349
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:39983
--webserver_port=43629
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 21:02:45 dist-test-slave-1dms krb5kdc[14211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:02:45.290588 14983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:45.266412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.1:43669)
0929 21:02:45.267235 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:45.267282 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 21:02:45.269180 (+ 1898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:02:45.270826 (+ 1646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:02:45.270833 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:02:45.271353 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:02:45.271890 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:45.271907 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:45.274424 (+ 2517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:45.274428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:45.276408 (+ 1980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:45.276427 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:45.277556 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:45.277566 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:45.277894 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:45.278658 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:02:45.278694 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:02:45.281470 (+ 2776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:02:45.286279 (+ 4809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:45.286287 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:45.286306 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:45.286781 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:45.287341 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:45.287344 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:45.287346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:45.287531 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:45.288553 (+ 1022us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:02:45.288559 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:02:45.288924 (+ 365us) client_negotiation.cc:769] Sending connection context
0929 21:02:45.289158 (+ 234us) client_negotiation.cc:240] Negotiation successful
0929 21:02:45.289558 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":359,"threads_started":1}
I20260929 21:02:45.291306 14982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:45.266283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43669 (local address 127.0.97.62:39983)
0929 21:02:45.266923 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:45.266939 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:45.267421 (+ 482us) server_negotiation.cc:407] Connection header received
0929 21:02:45.269773 (+ 2352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:45.269794 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:45.270259 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:45.270675 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:45.272134 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:45.274227 (+ 2093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:45.276572 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:45.277325 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:45.277464 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:45.281782 (+ 4318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:45.281797 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:45.281821 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:45.281883 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:45.286119 (+ 4236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:45.286946 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:45.286959 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:45.286971 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:45.287212 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:45.287640 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:45.287648 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:45.287654 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:45.288416 (+ 762us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:45.288616 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:45.289423 (+ 807us) server_negotiation.cc:299] Negotiation successful
0929 21:02:45.289914 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":272,"threads_started":1}
I20260929 21:02:45.293952 14980 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39983
I20260929 21:02:45.294320 14980 heartbeater.cc:464] Registering TS with master...
I20260929 21:02:45.295192 14980 heartbeater.cc:511] Master 127.0.97.62:39983 requested a full tablet report, sending...
I20260929 21:02:45.297459 14795 ts_manager.cc:194] Registered new tserver with Master: fa23bbba7a9042d8a58b0e95f853c442 (127.0.97.1:38137)
I20260929 21:02:45.298885 14795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43669
W20260929 21:02:45.603411 14984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:02:45.603700 14984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:02:45.603828 14984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:02:45.611716 14984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:02:45.611846 14984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:02:45.612021 14984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 21:02:45.622026 14984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:45349
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=43629
--tserver_master_addrs=127.0.97.62:39983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:02:45.623411 14984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:02:45.624801 14984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:02:45.638509 14992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:45.638646 14989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:45.638581 14990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:02:45.639724 14984 server_base.cc:1034] running on GCE node
I20260929 21:02:45.640458 14984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:02:45.641574 14984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:02:45.642802 14984 hybrid_clock.cc:648] HybridClock initialized: now 1790715765642741 us; error 64 us; skew 500 ppm
I20260929 21:02:45.643138 14984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:02:45 dist-test-slave-1dms krb5kdc[14211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:02:45.646402 14984 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 21:02:45.647881 14984 webserver.cc:460] Webserver started at http://127.0.97.2:43629/ using document root <none> and password file <none>
I20260929 21:02:45.648582 14984 fs_manager.cc:359] Metadata directory not provided
I20260929 21:02:45.648712 14984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:02:45.654627 14984 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 21:02:45.657789 14999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:45.659399 14984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:02:45.659559 14984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/wal
uuid: "059743faeb2246d5b6a5510034eaa7cb"
format_stamp: "Formatted at 2026-09-29 21:02:42 on dist-test-slave-1dms"
server_key: "35491f0d67ca90a535ce21725f820bdc"
server_key_iv: "7818b9b569bcdb1f151e9d699149c3a9"
server_key_version: "encryptionkey@0"
I20260929 21:02:45.660640 14984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:02:45.707722 14984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:02:45.711879 14984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:02:45.712154 14984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:02:45.713174 14984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:02:45.714501 14984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:02:45.714588 14984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:45.714694 14984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:02:45.714761 14984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:45.747298 14984 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45349
I20260929 21:02:45.747349 15111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45349 every 8 connection(s)
I20260929 21:02:45.748698 14984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
I20260929 21:02:45.755354 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 14984
I20260929 21:02:45.756075 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:40965
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:39983
--webserver_port=39693
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 21:02:45 dist-test-slave-1dms krb5kdc[14211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:02:45.772430 14982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:45.751429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48439 (local address 127.0.97.62:39983)
0929 21:02:45.751624 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:45.751630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:45.752881 (+ 1251us) server_negotiation.cc:407] Connection header received
0929 21:02:45.754601 (+ 1720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:45.754607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:45.754700 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:45.754804 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:45.756825 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:45.757913 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:45.760379 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:45.760795 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:45.760967 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:45.765215 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:45.765225 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:45.765233 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:45.765278 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:45.768927 (+ 3649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:45.769601 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:45.769609 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:45.769615 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:45.769772 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:45.770414 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:45.770423 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:45.770429 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:45.770856 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:45.771017 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:45.771707 (+ 690us) server_negotiation.cc:299] Negotiation successful
0929 21:02:45.771993 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 21:02:45.773104 15114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:45.751881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.2:48439)
0929 21:02:45.752684 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:45.752734 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 21:02:45.754358 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:02:45.755094 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:02:45.755107 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:02:45.755860 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:02:45.756627 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:45.756671 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:45.758080 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:45.758088 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:45.760239 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:45.760257 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:45.761069 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:45.761078 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:45.761392 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:45.762092 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:02:45.762125 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:02:45.765002 (+ 2877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:02:45.769063 (+ 4061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:45.769069 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:45.769082 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:45.769488 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:45.769893 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:45.769896 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:45.769899 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:45.770264 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:45.770995 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:02:45.771001 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:02:45.771328 (+ 327us) client_negotiation.cc:769] Sending connection context
0929 21:02:45.771563 (+ 235us) client_negotiation.cc:240] Negotiation successful
0929 21:02:45.771935 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":331,"threads_started":1}
I20260929 21:02:45.774619 15112 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39983
I20260929 21:02:45.774932 15112 heartbeater.cc:464] Registering TS with master...
I20260929 21:02:45.775612 15112 heartbeater.cc:511] Master 127.0.97.62:39983 requested a full tablet report, sending...
I20260929 21:02:45.777081 14795 ts_manager.cc:194] Registered new tserver with Master: 059743faeb2246d5b6a5510034eaa7cb (127.0.97.2:45349)
I20260929 21:02:45.777854 14795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48439
W20260929 21:02:46.081341 15115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:02:46.081662 15115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:02:46.081823 15115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:02:46.089123 15115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:02:46.089280 15115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:02:46.089447 15115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 21:02:46.099402 15115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:40965
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=39693
--tserver_master_addrs=127.0.97.62:39983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:02:46.100772 15115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:02:46.102105 15115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:02:46.115471 15123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:46.115541 15120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:46.115518 15121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:02:46.116322 15115 server_base.cc:1034] running on GCE node
I20260929 21:02:46.116958 15115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:02:46.117892 15115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:02:46.119103 15115 hybrid_clock.cc:648] HybridClock initialized: now 1790715766119063 us; error 40 us; skew 500 ppm
I20260929 21:02:46.119416 15115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:02:46 dist-test-slave-1dms krb5kdc[14211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:02:46.122510 15115 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 21:02:46.123971 15115 webserver.cc:460] Webserver started at http://127.0.97.3:39693/ using document root <none> and password file <none>
I20260929 21:02:46.124670 15115 fs_manager.cc:359] Metadata directory not provided
I20260929 21:02:46.124820 15115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:02:46.130131 15115 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260929 21:02:46.132926 15130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:46.134215 15115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 21:02:46.134383 15115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/wal
uuid: "c2dd5933f6464fa8a3ce2af782eacfb4"
format_stamp: "Formatted at 2026-09-29 21:02:43 on dist-test-slave-1dms"
server_key: "4b1b0d413af3cabbe47560980d78c6ad"
server_key_iv: "11ef40d280e0524898351c160bd39235"
server_key_version: "encryptionkey@0"
I20260929 21:02:46.135357 15115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:02:46.153578 15115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:02:46.157721 15115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:02:46.157985 15115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:02:46.158936 15115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:02:46.160239 15115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:02:46.160355 15115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:46.160471 15115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:02:46.160552 15115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:46.193539 15115 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40965
I20260929 21:02:46.193619 15242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40965 every 8 connection(s)
I20260929 21:02:46.194962 15115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
I20260929 21:02:46.202370 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 15115
Sep 29 21:02:46 dist-test-slave-1dms krb5kdc[14211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:02:46.216277 14982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:46.197589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51363 (local address 127.0.97.62:39983)
0929 21:02:46.197824 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:46.197833 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:46.198916 (+ 1083us) server_negotiation.cc:407] Connection header received
0929 21:02:46.200703 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:46.200709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:46.200802 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:46.200943 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:46.202401 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:46.203158 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:46.204886 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:46.205385 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:46.205504 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:46.209730 (+ 4226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:46.209735 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:46.209739 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:46.209767 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:46.212878 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:46.213889 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:46.213894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:46.213897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:46.214030 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:46.214485 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:46.214492 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:46.214495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:46.214845 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:46.214953 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:46.215899 (+ 946us) server_negotiation.cc:299] Negotiation successful
0929 21:02:46.216051 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 21:02:46.217911 15245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:46.198080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.3:51363)
0929 21:02:46.198754 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:46.198800 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 21:02:46.200438 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:02:46.201178 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:02:46.201186 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:02:46.201692 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:02:46.202225 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:46.202241 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:46.203329 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:46.203332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:46.204726 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:46.204738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:46.205601 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:46.205608 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:46.205850 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:46.206530 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:02:46.206552 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:02:46.209533 (+ 2981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:02:46.213151 (+ 3618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:46.213160 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:46.213177 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:46.213777 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:46.214159 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:02:46.214182 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:02:46.214185 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:02:46.214368 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:02:46.215002 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:02:46.215014 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:02:46.215495 (+ 481us) client_negotiation.cc:769] Sending connection context
0929 21:02:46.215832 (+ 337us) client_negotiation.cc:240] Negotiation successful
0929 21:02:46.216424 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":265,"threads_started":1}
I20260929 21:02:46.219406 15243 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39983
I20260929 21:02:46.219723 15243 heartbeater.cc:464] Registering TS with master...
I20260929 21:02:46.220626 15243 heartbeater.cc:511] Master 127.0.97.62:39983 requested a full tablet report, sending...
I20260929 21:02:46.222220 14795 ts_manager.cc:194] Registered new tserver with Master: c2dd5933f6464fa8a3ce2af782eacfb4 (127.0.97.3:40965)
I20260929 21:02:46.223040 14795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51363
I20260929 21:02:46.223578 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:46.207485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45394 (local address 127.0.97.62:39983)
0929 21:02:46.207995 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:46.208001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:46.208589 (+ 588us) server_negotiation.cc:407] Connection header received
0929 21:02:46.208774 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:46.208778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:46.208871 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:46.208996 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:46.210763 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:46.212004 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:46.213889 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:46.214444 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:46.214576 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:46.217780 (+ 3204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:46.217785 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:46.217789 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:46.217818 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:46.221010 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:46.221636 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:46.221641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:46.221644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:46.221763 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:46.222112 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:46.222121 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:46.222124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:46.222472 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:46.222739 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:46.223014 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 21:02:46.223227 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":859,"threads_started":1}
I20260929 21:02:46.225907 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 21:02:46.231828 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:46.226717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45396 (local address 127.0.97.62:39983)
0929 21:02:46.226868 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:46.226872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:46.227273 (+ 401us) server_negotiation.cc:407] Connection header received
0929 21:02:46.227492 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:46.227498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:46.227649 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:46.227768 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:02:46.228100 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:46.228717 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:46.230082 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:46.230490 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:46.230580 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:46.231002 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:02:46.231425 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:02:46.231528 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:46.231595 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 21:02:46.231650 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 21:02:46.232654 14795 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:45396:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 21:02:46.240017 14795 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 21:02:46.261577 15257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:46.253109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40965 (local address 127.0.0.1:57454)
0929 21:02:46.253590 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:46.253608 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 21:02:46.253708 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:02:46.254803 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:02:46.254808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:02:46.254832 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:02:46.255145 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:46.255158 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:46.257212 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:46.257217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:46.259057 (+ 1840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:46.259069 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:46.260442 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:46.260449 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:46.260631 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:46.260685 (+ 54us) client_negotiation.cc:769] Sending connection context
0929 21:02:46.260853 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 21:02:46.261063 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":19,"thread_start_us":840,"threads_started":1}
I20260929 21:02:46.261865 15254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:46.250737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45349 (local address 127.0.0.1:35374)
0929 21:02:46.251893 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:46.251932 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 21:02:46.252078 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:02:46.254031 (+ 1953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:02:46.254042 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:02:46.254084 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:02:46.254581 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:46.254596 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:46.256508 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:46.256513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:46.258969 (+ 2456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:46.258987 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:46.260902 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:46.261043 (+ 141us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:46.261228 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:46.261263 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 21:02:46.261377 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 21:02:46.261512 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":934,"thread_start_us":1964,"threads_started":1}
I20260929 21:02:46.261942 15255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:46.251020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35374 (local address 127.0.97.2:45349)
0929 21:02:46.252899 (+ 1879us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:46.252911 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:46.252956 (+ 45us) server_negotiation.cc:407] Connection header received
0929 21:02:46.253084 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:46.253096 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:46.253415 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:46.253640 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:02:46.254775 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:46.256374 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:46.259149 (+ 2775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:46.260655 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:46.260778 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:46.261031 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:46.261433 (+ 402us) server_negotiation.cc:299] Negotiation successful
0929 21:02:46.261594 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1439,"thread_start_us":1297,"threads_started":1}
I20260929 21:02:46.262651 15258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:46.253662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57454 (local address 127.0.97.3:40965)
0929 21:02:46.254345 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:46.254350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:46.254374 (+ 24us) server_negotiation.cc:407] Connection header received
0929 21:02:46.254444 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:46.254448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:46.254601 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:46.255367 (+ 766us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:02:46.255608 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:46.257039 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:46.259255 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:46.260215 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:46.260383 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:46.260624 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:46.262145 (+ 1521us) server_negotiation.cc:299] Negotiation successful
0929 21:02:46.262324 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":441,"threads_started":1}
I20260929 21:02:46.265949 15178 tablet_service.cc:1467] Processing CreateTablet for tablet d802e37f88c84d1e8a34b43ab087e0ca (DEFAULT_TABLE table=test-table [id=cae4c9553e8f476ab2c9c87e53557f8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:02:46.265951 15047 tablet_service.cc:1467] Processing CreateTablet for tablet d802e37f88c84d1e8a34b43ab087e0ca (DEFAULT_TABLE table=test-table [id=cae4c9553e8f476ab2c9c87e53557f8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:02:46.267076 15178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d802e37f88c84d1e8a34b43ab087e0ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:02:46.267242 15047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d802e37f88c84d1e8a34b43ab087e0ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:02:46.271595 15253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:46.250904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38137 (local address 127.0.0.1:39780)
0929 21:02:46.252985 (+ 2081us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:46.253004 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 21:02:46.253109 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:02:46.257010 (+ 3901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:02:46.257014 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:02:46.257039 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:02:46.257583 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:46.257595 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:46.259397 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:46.259402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:46.261401 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:46.261414 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:46.270682 (+ 9268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:46.270690 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:46.270900 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:46.270939 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 21:02:46.271047 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 21:02:46.271199 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1955,"mutex_wait_us":2,"thread_start_us":650,"threads_started":1}
I20260929 21:02:46.271608 15256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:46.251399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39780 (local address 127.0.97.1:38137)
0929 21:02:46.256370 (+ 4971us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:46.256376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:46.256397 (+ 21us) server_negotiation.cc:407] Connection header received
0929 21:02:46.256495 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:46.256502 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:46.256741 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:46.257045 (+ 304us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:02:46.257771 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:46.259245 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:46.268967 (+ 9722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:46.270432 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:46.270569 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:46.270782 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:46.271107 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 21:02:46.271276 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4820,"thread_start_us":4721,"threads_started":1}
I20260929 21:02:46.275712 14915 tablet_service.cc:1467] Processing CreateTablet for tablet d802e37f88c84d1e8a34b43ab087e0ca (DEFAULT_TABLE table=test-table [id=cae4c9553e8f476ab2c9c87e53557f8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:02:46.277164 14915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d802e37f88c84d1e8a34b43ab087e0ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:02:46.285784 15260 tablet_bootstrap.cc:492] T d802e37f88c84d1e8a34b43ab087e0ca P 059743faeb2246d5b6a5510034eaa7cb: Bootstrap starting.
I20260929 21:02:46.289630 15259 tablet_bootstrap.cc:492] T d802e37f88c84d1e8a34b43ab087e0ca P c2dd5933f6464fa8a3ce2af782eacfb4: Bootstrap starting.
I20260929 21:02:46.290536 15260 tablet_bootstrap.cc:654] T d802e37f88c84d1e8a34b43ab087e0ca P 059743faeb2246d5b6a5510034eaa7cb: Neither blocks nor log segments found. Creating new log.
I20260929 21:02:46.291553 15260 log.cc:824] T d802e37f88c84d1e8a34b43ab087e0ca P 059743faeb2246d5b6a5510034eaa7cb: Log is configured to *not* fsync() on all Append() calls
I20260929 21:02:46.293093 15259 tablet_bootstrap.cc:654] T d802e37f88c84d1e8a34b43ab087e0ca P c2dd5933f6464fa8a3ce2af782eacfb4: Neither blocks nor log segments found. Creating new log.
I20260929 21:02:46.293889 15262 tablet_bootstrap.cc:492] T d802e37f88c84d1e8a34b43ab087e0ca P fa23bbba7a9042d8a58b0e95f853c442: Bootstrap starting.
I20260929 21:02:46.294389 15259 log.cc:824] T d802e37f88c84d1e8a34b43ab087e0ca P c2dd5933f6464fa8a3ce2af782eacfb4: Log is configured to *not* fsync() on all Append() calls
I20260929 21:02:46.294968 15260 tablet_bootstrap.cc:492] T d802e37f88c84d1e8a34b43ab087e0ca P 059743faeb2246d5b6a5510034eaa7cb: No bootstrap required, opened a new log
I20260929 21:02:46.295243 15260 ts_tablet_manager.cc:1397] T d802e37f88c84d1e8a34b43ab087e0ca P 059743faeb2246d5b6a5510034eaa7cb: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20260929 21:02:46.297380 15259 tablet_bootstrap.cc:492] T d802e37f88c84d1e8a34b43ab087e0ca P c2dd5933f6464fa8a3ce2af782eacfb4: No bootstrap required, opened a new log
I20260929 21:02:46.297601 15262 tablet_bootstrap.cc:654] T d802e37f88c84d1e8a34b43ab087e0ca P fa23bbba7a9042d8a58b0e95f853c442: Neither blocks nor log segments found. Creating new log.
I20260929 21:02:46.297631 15259 ts_tablet_manager.cc:1397] T d802e37f88c84d1e8a34b43ab087e0ca P c2dd5933f6464fa8a3ce2af782eacfb4: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 21:02:46.298822 15262 log.cc:824] T d802e37f88c84d1e8a34b43ab087e0ca P fa23bbba7a9042d8a58b0e95f853c442: Log is configured to *not* fsync() on all Append() calls
I20260929 21:02:46.301978 15262 tablet_bootstrap.cc:492] T d802e37f88c84d1e8a34b43ab087e0ca P fa23bbba7a9042d8a58b0e95f853c442: No bootstrap required, opened a new log
I20260929 21:02:46.302255 15262 ts_tablet_manager.cc:1397] T d802e37f88c84d1e8a34b43ab087e0ca P fa23bbba7a9042d8a58b0e95f853c442: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 21:02:46.302393 14980 heartbeater.cc:503] Master 127.0.97.62:39983 was elected leader, sending a full tablet report...
I20260929 21:02:46.305907 15260 raft_consensus.cc:348] T d802e37f88c84d1e8a34b43ab087e0ca P 059743faeb2246d5b6a5510034eaa7cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059743faeb2246d5b6a5510034eaa7cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45349 } } peers { permanent_uuid: "fa23bbba7a9042d8a58b0e95f853c442" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38137 } } peers { permanent_uuid: "c2dd5933f6464fa8a3ce2af782eacfb4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40965 } }
I20260929 21:02:46.306380 15260 raft_consensus.cc:374] T d802e37f88c84d1e8a34b43ab087e0ca P 059743faeb2246d5b6a5510034eaa7cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:02:46.306475 15260 raft_consensus.cc:729] T d802e37f88c84d1e8a34b43ab087e0ca P 059743faeb2246d5b6a5510034eaa7cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059743faeb2246d5b6a5510034eaa7cb, State: Initialized, Role: FOLLOWER
I20260929 21:02:46.306349 15259 raft_consensus.cc:348] T d802e37f88c84d1e8a34b43ab087e0ca P c2dd5933f6464fa8a3ce2af782eacfb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059743faeb2246d5b6a5510034eaa7cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45349 } } peers { permanent_uuid: "fa23bbba7a9042d8a58b0e95f853c442" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38137 } } peers { permanent_uuid: "c2dd5933f6464fa8a3ce2af782eacfb4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40965 } }
I20260929 21:02:46.306769 15259 raft_consensus.cc:374] T d802e37f88c84d1e8a34b43ab087e0ca P c2dd5933f6464fa8a3ce2af782eacfb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:02:46.306871 15259 raft_consensus.cc:729] T d802e37f88c84d1e8a34b43ab087e0ca P c2dd5933f6464fa8a3ce2af782eacfb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2dd5933f6464fa8a3ce2af782eacfb4, State: Initialized, Role: FOLLOWER
I20260929 21:02:46.307212 15260 consensus_queue.cc:260] T d802e37f88c84d1e8a34b43ab087e0ca P 059743faeb2246d5b6a5510034eaa7cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059743faeb2246d5b6a5510034eaa7cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45349 } } peers { permanent_uuid: "fa23bbba7a9042d8a58b0e95f853c442" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38137 } } peers { permanent_uuid: "c2dd5933f6464fa8a3ce2af782eacfb4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40965 } }
I20260929 21:02:46.307562 15259 consensus_queue.cc:260] T d802e37f88c84d1e8a34b43ab087e0ca P c2dd5933f6464fa8a3ce2af782eacfb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059743faeb2246d5b6a5510034eaa7cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45349 } } peers { permanent_uuid: "fa23bbba7a9042d8a58b0e95f853c442" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38137 } } peers { permanent_uuid: "c2dd5933f6464fa8a3ce2af782eacfb4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40965 } }
I20260929 21:02:46.313783 15112 heartbeater.cc:503] Master 127.0.97.62:39983 was elected leader, sending a full tablet report...
I20260929 21:02:46.314358 15243 heartbeater.cc:503] Master 127.0.97.62:39983 was elected leader, sending a full tablet report...
I20260929 21:02:46.314569 15262 raft_consensus.cc:348] T d802e37f88c84d1e8a34b43ab087e0ca P fa23bbba7a9042d8a58b0e95f853c442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059743faeb2246d5b6a5510034eaa7cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45349 } } peers { permanent_uuid: "fa23bbba7a9042d8a58b0e95f853c442" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38137 } } peers { permanent_uuid: "c2dd5933f6464fa8a3ce2af782eacfb4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40965 } }
I20260929 21:02:46.315042 15262 raft_consensus.cc:374] T d802e37f88c84d1e8a34b43ab087e0ca P fa23bbba7a9042d8a58b0e95f853c442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:02:46.315086 15259 ts_tablet_manager.cc:1428] T d802e37f88c84d1e8a34b43ab087e0ca P c2dd5933f6464fa8a3ce2af782eacfb4: Time spent starting tablet: real 0.017s user 0.005s sys 0.012s
I20260929 21:02:46.315155 15262 raft_consensus.cc:729] T d802e37f88c84d1e8a34b43ab087e0ca P fa23bbba7a9042d8a58b0e95f853c442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa23bbba7a9042d8a58b0e95f853c442, State: Initialized, Role: FOLLOWER
I20260929 21:02:46.315816 15262 consensus_queue.cc:260] T d802e37f88c84d1e8a34b43ab087e0ca P fa23bbba7a9042d8a58b0e95f853c442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059743faeb2246d5b6a5510034eaa7cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45349 } } peers { permanent_uuid: "fa23bbba7a9042d8a58b0e95f853c442" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38137 } } peers { permanent_uuid: "c2dd5933f6464fa8a3ce2af782eacfb4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40965 } }
I20260929 21:02:46.316710 15260 ts_tablet_manager.cc:1428] T d802e37f88c84d1e8a34b43ab087e0ca P 059743faeb2246d5b6a5510034eaa7cb: Time spent starting tablet: real 0.021s user 0.005s sys 0.013s
I20260929 21:02:46.318511 15262 ts_tablet_manager.cc:1428] T d802e37f88c84d1e8a34b43ab087e0ca P fa23bbba7a9042d8a58b0e95f853c442: Time spent starting tablet: real 0.016s user 0.013s sys 0.004s
I20260929 21:02:46.366487 15265 raft_consensus.cc:482] T d802e37f88c84d1e8a34b43ab087e0ca P 059743faeb2246d5b6a5510034eaa7cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 21:02:46.366799 15265 raft_consensus.cc:504] T d802e37f88c84d1e8a34b43ab087e0ca P 059743faeb2246d5b6a5510034eaa7cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059743faeb2246d5b6a5510034eaa7cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45349 } } peers { permanent_uuid: "fa23bbba7a9042d8a58b0e95f853c442" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38137 } } peers { permanent_uuid: "c2dd5933f6464fa8a3ce2af782eacfb4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40965 } }
I20260929 21:02:46.368484 15265 leader_election.cc:288] T d802e37f88c84d1e8a34b43ab087e0ca P 059743faeb2246d5b6a5510034eaa7cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa23bbba7a9042d8a58b0e95f853c442 (127.0.97.1:38137), c2dd5933f6464fa8a3ce2af782eacfb4 (127.0.97.3:40965)
I20260929 21:02:46.375204 15258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:46.369128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44187 (local address 127.0.97.3:40965)
0929 21:02:46.369389 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:46.369399 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:46.370028 (+ 629us) server_negotiation.cc:407] Connection header received
0929 21:02:46.370207 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:46.370211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:46.370269 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:46.370355 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:02:46.371018 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:46.371717 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:46.373496 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:46.374251 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:46.374397 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:46.374509 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:46.374830 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 21:02:46.374948 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 21:02:46.375262 15256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:46.368966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47253 (local address 127.0.97.1:38137)
0929 21:02:46.369435 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:46.369444 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:46.369964 (+ 520us) server_negotiation.cc:407] Connection header received
0929 21:02:46.370205 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:46.370209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:46.370268 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:46.370364 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:02:46.370963 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:46.371650 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:46.373508 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:46.374295 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:46.374451 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:46.374561 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:46.374866 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 21:02:46.374989 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258}
I20260929 21:02:46.375356 15268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:46.368692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38137 (local address 127.0.97.2:47253)
0929 21:02:46.369857 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:46.369873 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 21:02:46.370041 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:02:46.370381 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:02:46.370385 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:02:46.370408 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:02:46.370821 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:46.370833 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:46.371777 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:46.371785 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:46.373355 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:46.373367 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:46.374522 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:46.374527 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:46.374700 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:46.374727 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 21:02:46.374840 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 21:02:46.374964 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1034,"thread_start_us":793,"threads_started":1}
I20260929 21:02:46.375763 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:46.368773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40965 (local address 127.0.97.2:44187)
0929 21:02:46.369947 (+ 1174us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:46.369961 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 21:02:46.370067 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:02:46.370387 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:02:46.370391 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:02:46.370474 (+ 83us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:02:46.370849 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:46.370858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:46.371859 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:46.371869 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:46.373357 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:02:46.373367 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:02:46.374477 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:02:46.374483 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:02:46.374646 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:46.374669 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 21:02:46.374776 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 21:02:46.374913 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1051,"mutex_wait_us":2,"spinlock_wait_cycles":25856,"thread_start_us":838,"threads_started":1}
I20260929 21:02:46.376379 14935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d802e37f88c84d1e8a34b43ab087e0ca" candidate_uuid: "059743faeb2246d5b6a5510034eaa7cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa23bbba7a9042d8a58b0e95f853c442" is_pre_election: true
I20260929 21:02:46.376700 15198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d802e37f88c84d1e8a34b43ab087e0ca" candidate_uuid: "059743faeb2246d5b6a5510034eaa7cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2dd5933f6464fa8a3ce2af782eacfb4" is_pre_election: true
I20260929 21:02:46.376942 14935 raft_consensus.cc:2454] T d802e37f88c84d1e8a34b43ab087e0ca P fa23bbba7a9042d8a58b0e95f853c442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059743faeb2246d5b6a5510034eaa7cb in term 0.
I20260929 21:02:46.377238 15198 raft_consensus.cc:2454] T d802e37f88c84d1e8a34b43ab087e0ca P c2dd5933f6464fa8a3ce2af782eacfb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059743faeb2246d5b6a5510034eaa7cb in term 0.
I20260929 21:02:46.377704 15000 leader_election.cc:302] T d802e37f88c84d1e8a34b43ab087e0ca P 059743faeb2246d5b6a5510034eaa7cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059743faeb2246d5b6a5510034eaa7cb, fa23bbba7a9042d8a58b0e95f853c442; no voters:
I20260929 21:02:46.378101 15265 raft_consensus.cc:2781] T d802e37f88c84d1e8a34b43ab087e0ca P 059743faeb2246d5b6a5510034eaa7cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 21:02:46.378262 15265 raft_consensus.cc:482] T d802e37f88c84d1e8a34b43ab087e0ca P 059743faeb2246d5b6a5510034eaa7cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 21:02:46.378365 15265 raft_consensus.cc:3037] T d802e37f88c84d1e8a34b43ab087e0ca P 059743faeb2246d5b6a5510034eaa7cb [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:02:46.380681 15265 raft_consensus.cc:504] T d802e37f88c84d1e8a34b43ab087e0ca P 059743faeb2246d5b6a5510034eaa7cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059743faeb2246d5b6a5510034eaa7cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45349 } } peers { permanent_uuid: "fa23bbba7a9042d8a58b0e95f853c442" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38137 } } peers { permanent_uuid: "c2dd5933f6464fa8a3ce2af782eacfb4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40965 } }
I20260929 21:02:46.381397 15265 leader_election.cc:288] T d802e37f88c84d1e8a34b43ab087e0ca P 059743faeb2246d5b6a5510034eaa7cb [CANDIDATE]: Term 1 election: Requested vote from peers fa23bbba7a9042d8a58b0e95f853c442 (127.0.97.1:38137), c2dd5933f6464fa8a3ce2af782eacfb4 (127.0.97.3:40965)
I20260929 21:02:46.381836 14935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d802e37f88c84d1e8a34b43ab087e0ca" candidate_uuid: "059743faeb2246d5b6a5510034eaa7cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa23bbba7a9042d8a58b0e95f853c442"
I20260929 21:02:46.381932 15198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d802e37f88c84d1e8a34b43ab087e0ca" candidate_uuid: "059743faeb2246d5b6a5510034eaa7cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2dd5933f6464fa8a3ce2af782eacfb4"
I20260929 21:02:46.382059 14935 raft_consensus.cc:3037] T d802e37f88c84d1e8a34b43ab087e0ca P fa23bbba7a9042d8a58b0e95f853c442 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:02:46.382117 15198 raft_consensus.cc:3037] T d802e37f88c84d1e8a34b43ab087e0ca P c2dd5933f6464fa8a3ce2af782eacfb4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:02:46.385314 14935 raft_consensus.cc:2454] T d802e37f88c84d1e8a34b43ab087e0ca P fa23bbba7a9042d8a58b0e95f853c442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059743faeb2246d5b6a5510034eaa7cb in term 1.
I20260929 21:02:46.385321 15198 raft_consensus.cc:2454] T d802e37f88c84d1e8a34b43ab087e0ca P c2dd5933f6464fa8a3ce2af782eacfb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059743faeb2246d5b6a5510034eaa7cb in term 1.
I20260929 21:02:46.385708 15002 leader_election.cc:302] T d802e37f88c84d1e8a34b43ab087e0ca P 059743faeb2246d5b6a5510034eaa7cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059743faeb2246d5b6a5510034eaa7cb, c2dd5933f6464fa8a3ce2af782eacfb4; no voters:
I20260929 21:02:46.386052 15265 raft_consensus.cc:2781] T d802e37f88c84d1e8a34b43ab087e0ca P 059743faeb2246d5b6a5510034eaa7cb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:02:46.386510 15265 raft_consensus.cc:686] T d802e37f88c84d1e8a34b43ab087e0ca P 059743faeb2246d5b6a5510034eaa7cb [term 1 LEADER]: Becoming Leader. State: Replica: 059743faeb2246d5b6a5510034eaa7cb, State: Running, Role: LEADER
I20260929 21:02:46.386968 15265 consensus_queue.cc:237] T d802e37f88c84d1e8a34b43ab087e0ca P 059743faeb2246d5b6a5510034eaa7cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059743faeb2246d5b6a5510034eaa7cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45349 } } peers { permanent_uuid: "fa23bbba7a9042d8a58b0e95f853c442" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38137 } } peers { permanent_uuid: "c2dd5933f6464fa8a3ce2af782eacfb4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40965 } }
I20260929 21:02:46.391709 14793 catalog_manager.cc:5495] T d802e37f88c84d1e8a34b43ab087e0ca P 059743faeb2246d5b6a5510034eaa7cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 059743faeb2246d5b6a5510034eaa7cb (127.0.97.2). New cstate: current_term: 1 leader_uuid: "059743faeb2246d5b6a5510034eaa7cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059743faeb2246d5b6a5510034eaa7cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa23bbba7a9042d8a58b0e95f853c442" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2dd5933f6464fa8a3ce2af782eacfb4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40965 } health_report { overall_health: UNKNOWN } } }
I20260929 21:02:46.423418 15255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:46.418208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35382 (local address 127.0.97.2:45349)
0929 21:02:46.418465 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:46.418473 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:46.418491 (+ 18us) server_negotiation.cc:407] Connection header received
0929 21:02:46.418578 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:46.418582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:46.418652 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:46.418796 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:02:46.419163 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:46.419855 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:46.421367 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:46.421767 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:46.421894 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:46.422363 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:02:46.422851 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:02:46.423016 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:46.423075 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 21:02:46.423153 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 21:02:46.432013 15198 raft_consensus.cc:1261] T d802e37f88c84d1e8a34b43ab087e0ca P c2dd5933f6464fa8a3ce2af782eacfb4 [term 1 FOLLOWER]: Refusing update from remote peer 059743faeb2246d5b6a5510034eaa7cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:02:46.432013 14935 raft_consensus.cc:1261] T d802e37f88c84d1e8a34b43ab087e0ca P fa23bbba7a9042d8a58b0e95f853c442 [term 1 FOLLOWER]: Refusing update from remote peer 059743faeb2246d5b6a5510034eaa7cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:02:46.433116 15270 consensus_queue.cc:1035] T d802e37f88c84d1e8a34b43ab087e0ca P 059743faeb2246d5b6a5510034eaa7cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2dd5933f6464fa8a3ce2af782eacfb4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:02:46.433530 15265 consensus_queue.cc:1035] T d802e37f88c84d1e8a34b43ab087e0ca P 059743faeb2246d5b6a5510034eaa7cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa23bbba7a9042d8a58b0e95f853c442" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:02:46.444737 15272 mvcc.cc:205] Tried to move back new op lower bound from 7334771779294306304 to 7334771779125100544. Current Snapshot: MvccSnapshot[applied={T|T < 7334771779294306304}]
W20260929 21:02:46.446647 15244 tablet.cc:2369] T d802e37f88c84d1e8a34b43ab087e0ca P c2dd5933f6464fa8a3ce2af782eacfb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 21:02:46.449497 15274 mvcc.cc:205] Tried to move back new op lower bound from 7334771779294306304 to 7334771779125100544. Current Snapshot: MvccSnapshot[applied={T|T < 7334771779294306304}]
I20260929 21:02:46.458489 14795 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:45396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 21:02:46.458766 14795 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:45396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 21:02:46.462137 14795 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16: Sending DeleteTablet for 3 replicas of tablet d802e37f88c84d1e8a34b43ab087e0ca
I20260929 21:02:46.463423 15047 tablet_service.cc:1514] Processing DeleteTablet for tablet d802e37f88c84d1e8a34b43ab087e0ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:02:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35374
I20260929 21:02:46.463693 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 14832
I20260929 21:02:46.463848 15178 tablet_service.cc:1514] Processing DeleteTablet for tablet d802e37f88c84d1e8a34b43ab087e0ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:02:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57454
I20260929 21:02:46.463840 14915 tablet_service.cc:1514] Processing DeleteTablet for tablet d802e37f88c84d1e8a34b43ab087e0ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:02:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39780
I20260929 21:02:46.465138 15283 tablet_replica.cc:330] T d802e37f88c84d1e8a34b43ab087e0ca P fa23bbba7a9042d8a58b0e95f853c442: stopping tablet replica
I20260929 21:02:46.465873 15283 raft_consensus.cc:2229] T d802e37f88c84d1e8a34b43ab087e0ca P fa23bbba7a9042d8a58b0e95f853c442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:02:46.466609 15283 raft_consensus.cc:2258] T d802e37f88c84d1e8a34b43ab087e0ca P fa23bbba7a9042d8a58b0e95f853c442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:02:46.474228 15282 tablet_replica.cc:330] T d802e37f88c84d1e8a34b43ab087e0ca P c2dd5933f6464fa8a3ce2af782eacfb4: stopping tablet replica
I20260929 21:02:46.474453 15283 ts_tablet_manager.cc:1905] T d802e37f88c84d1e8a34b43ab087e0ca P fa23bbba7a9042d8a58b0e95f853c442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:02:46.474710 15282 raft_consensus.cc:2229] T d802e37f88c84d1e8a34b43ab087e0ca P c2dd5933f6464fa8a3ce2af782eacfb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:02:46.475243 15282 raft_consensus.cc:2258] T d802e37f88c84d1e8a34b43ab087e0ca P c2dd5933f6464fa8a3ce2af782eacfb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:02:46.478555 15281 tablet_replica.cc:330] T d802e37f88c84d1e8a34b43ab087e0ca P 059743faeb2246d5b6a5510034eaa7cb: stopping tablet replica
I20260929 21:02:46.478988 15281 raft_consensus.cc:2229] T d802e37f88c84d1e8a34b43ab087e0ca P 059743faeb2246d5b6a5510034eaa7cb [term 1 LEADER]: Raft consensus shutting down.
I20260929 21:02:46.479513 15282 ts_tablet_manager.cc:1905] T d802e37f88c84d1e8a34b43ab087e0ca P c2dd5933f6464fa8a3ce2af782eacfb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:02:46.480055 15281 raft_consensus.cc:2258] T d802e37f88c84d1e8a34b43ab087e0ca P 059743faeb2246d5b6a5510034eaa7cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:02:46.484728 15256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:46.464789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39788 (local address 127.0.97.1:38137)
0929 21:02:46.465587 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:46.465595 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:46.465618 (+ 23us) server_negotiation.cc:407] Connection header received
0929 21:02:46.465689 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:46.465697 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:46.465800 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:46.465921 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:46.468427 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:46.470030 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:46.472574 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:46.473269 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:46.473422 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:46.477322 (+ 3900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:46.477338 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:46.477358 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:46.477410 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:46.481432 (+ 4022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:46.482288 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:46.482305 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:46.482309 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:46.482468 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:46.482866 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:46.482871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:46.482874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:46.483412 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:46.483698 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:46.484038 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 21:02:46.484325 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611}
I20260929 21:02:46.485584 14975 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:02:46.486343 15281 ts_tablet_manager.cc:1905] T d802e37f88c84d1e8a34b43ab087e0ca P 059743faeb2246d5b6a5510034eaa7cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:02:46.487835 15282 ts_tablet_manager.cc:1918] T d802e37f88c84d1e8a34b43ab087e0ca P c2dd5933f6464fa8a3ce2af782eacfb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:02:46.488072 15282 log.cc:1196] T d802e37f88c84d1e8a34b43ab087e0ca P c2dd5933f6464fa8a3ce2af782eacfb4: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/wal/wals/d802e37f88c84d1e8a34b43ab087e0ca
I20260929 21:02:46.488680 15282 ts_tablet_manager.cc:1939] T d802e37f88c84d1e8a34b43ab087e0ca P c2dd5933f6464fa8a3ce2af782eacfb4: Deleting consensus metadata
I20260929 21:02:46.490626 14781 catalog_manager.cc:4841] TS c2dd5933f6464fa8a3ce2af782eacfb4 (127.0.97.3:40965): tablet d802e37f88c84d1e8a34b43ab087e0ca (table test-table [id=cae4c9553e8f476ab2c9c87e53557f8e]) successfully deleted
I20260929 21:02:46.495236 15281 ts_tablet_manager.cc:1918] T d802e37f88c84d1e8a34b43ab087e0ca P 059743faeb2246d5b6a5510034eaa7cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:02:46.495411 15281 log.cc:1196] T d802e37f88c84d1e8a34b43ab087e0ca P 059743faeb2246d5b6a5510034eaa7cb: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/wal/wals/d802e37f88c84d1e8a34b43ab087e0ca
I20260929 21:02:46.495978 15281 ts_tablet_manager.cc:1939] T d802e37f88c84d1e8a34b43ab087e0ca P 059743faeb2246d5b6a5510034eaa7cb: Deleting consensus metadata
I20260929 21:02:46.497761 14781 catalog_manager.cc:4841] TS 059743faeb2246d5b6a5510034eaa7cb (127.0.97.2:45349): tablet d802e37f88c84d1e8a34b43ab087e0ca (table test-table [id=cae4c9553e8f476ab2c9c87e53557f8e]) successfully deleted
I20260929 21:02:46.584007 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 14832
I20260929 21:02:46.607808 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 14984
W20260929 21:02:46.607929 14779 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:38137 (error 108)
I20260929 21:02:46.611462 15253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:46.611058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38137 (local address 127.0.0.1:39802)
0929 21:02:46.611228 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:46.611331 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 21:02:46.611817 14779 catalog_manager.cc:4584] TS fa23bbba7a9042d8a58b0e95f853c442 (127.0.97.1:38137): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d802e37f88c84d1e8a34b43ab087e0ca: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38137: connect: Connection refused (error 111)
I20260929 21:02:46.624511 15255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:46.608644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35396 (local address 127.0.97.2:45349)
0929 21:02:46.609539 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:46.609548 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:46.609601 (+ 53us) server_negotiation.cc:407] Connection header received
0929 21:02:46.609687 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:46.609695 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:46.609797 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:46.609928 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:46.611710 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:46.612625 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:46.614615 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:46.615049 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:46.615164 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:46.618497 (+ 3333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:46.618507 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:46.618521 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:46.618552 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:46.621855 (+ 3303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:46.622657 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:46.622663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:46.622666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:46.622822 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:46.623184 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:46.623187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:46.623189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:46.623504 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:46.623739 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:46.624042 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 21:02:46.624191 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712}
I20260929 21:02:46.625289 15107 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:02:46.665449 15253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:46.664968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38137 (local address 127.0.0.1:39804)
0929 21:02:46.665176 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:46.665279 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 21:02:46.666440 15253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:46.666125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38137 (local address 127.0.0.1:39820)
0929 21:02:46.666242 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:46.666330 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 21:02:46.727098 15253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:46.726631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38137 (local address 127.0.0.1:39832)
0929 21:02:46.726837 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:46.726930 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 21:02:46.728111 15253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:46.727770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38137 (local address 127.0.0.1:39838)
0929 21:02:46.727888 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:46.727966 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 21:02:46.736907 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 14984
I20260929 21:02:46.756150 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 15115
I20260929 21:02:46.768714 15258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:46.756854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57466 (local address 127.0.97.3:40965)
0929 21:02:46.757030 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 21:02:46.757035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:02:46.757052 (+ 17us) server_negotiation.cc:407] Connection header received
0929 21:02:46.757179 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:02:46.757183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:02:46.757256 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:02:46.757338 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:02:46.758510 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:46.759207 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:46.760397 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:02:46.760769 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:02:46.760949 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:02:46.763561 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:02:46.763576 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:02:46.763592 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:02:46.763626 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:02:46.766191 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:46.766827 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:46.766834 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:46.766836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:46.766977 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:02:46.767270 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:02:46.767273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:02:46.767275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:02:46.767639 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:02:46.767846 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 21:02:46.768090 (+ 244us) server_negotiation.cc:299] Negotiation successful
0929 21:02:46.768253 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 21:02:46.769344 15238 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:02:46.808544 15253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:46.807972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38137 (local address 127.0.0.1:39854)
0929 21:02:46.808197 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:46.808312 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 21:02:46.809713 15253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:46.809244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38137 (local address 127.0.0.1:39862)
0929 21:02:46.809462 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:46.809560 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260929 21:02:46.859019 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 15115
I20260929 21:02:46.876046 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 14762
I20260929 21:02:46.877225 14825 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:02:46.990492 15253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:46.990048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38137 (local address 127.0.0.1:39876)
0929 21:02:46.990205 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:46.990287 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 21:02:46.990530 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 14762
I20260929 21:02:47.013079 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39983
--webserver_interface=127.0.97.62
--webserver_port=39115
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39983
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:02:47.332436 15289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:02:47.332793 15289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:02:47.332962 15289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:02:47.340118 15289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:02:47.340268 15289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:02:47.340370 15289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:02:47.340474 15289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:02:47.340549 15289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:02:47.350262 15289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=39115
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:02:47.351548 15289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:02:47.352826 15289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:02:47.364324 15294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:47.364404 15297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:47.364369 15295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:02:47.365589 15289 server_base.cc:1034] running on GCE node
I20260929 21:02:47.366334 15289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:02:47.367796 15289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:02:47.369022 15289 hybrid_clock.cc:648] HybridClock initialized: now 1790715767368983 us; error 50 us; skew 500 ppm
I20260929 21:02:47.369401 15289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:02:47 dist-test-slave-1dms krb5kdc[14211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:02:47.372395 15289 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 21:02:47.373821 15289 webserver.cc:460] Webserver started at http://127.0.97.62:39115/ using document root <none> and password file <none>
I20260929 21:02:47.374444 15289 fs_manager.cc:359] Metadata directory not provided
I20260929 21:02:47.374547 15289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:02:47.379616 15289 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 21:02:47.382218 15304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:47.383474 15289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 21:02:47.383647 15289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "bda88db6437c46fda958bcb092445d16"
format_stamp: "Formatted at 2026-09-29 21:02:41 on dist-test-slave-1dms"
server_key: "1eff2d0bff18c77852696b1f5d453896"
server_key_iv: "3593af8c53c54a3d5497dd17e8de5696"
server_key_version: "encryptionkey@0"
I20260929 21:02:47.384589 15289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:02:47.397926 15289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:02:47.401661 15289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:02:47.401927 15289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:02:47.419628 15289 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39983
I20260929 21:02:47.419660 15355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39983 every 8 connection(s)
I20260929 21:02:47.421075 15289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:02:47.423424 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 15289
I20260929 21:02:47.424382 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:38137
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:39983
--webserver_port=42689
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 21:02:47.430174 15356 sys_catalog.cc:263] Verifying existing consensus state
I20260929 21:02:47.434460 15356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16: Bootstrap starting.
I20260929 21:02:47.454183 15356 log.cc:824] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16: Log is configured to *not* fsync() on all Append() calls
I20260929 21:02:47.469743 15356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 21:02:47.470255 15356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16: Bootstrap complete.
I20260929 21:02:47.479097 15356 raft_consensus.cc:348] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda88db6437c46fda958bcb092445d16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39983 } }
I20260929 21:02:47.479960 15356 raft_consensus.cc:729] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: bda88db6437c46fda958bcb092445d16, State: Initialized, Role: FOLLOWER
I20260929 21:02:47.480578 15356 consensus_queue.cc:260] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda88db6437c46fda958bcb092445d16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39983 } }
I20260929 21:02:47.480793 15356 raft_consensus.cc:388] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:02:47.480916 15356 raft_consensus.cc:482] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:02:47.481101 15356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [term 2 FOLLOWER]: Advancing to term 3
I20260929 21:02:47.484704 15356 raft_consensus.cc:504] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda88db6437c46fda958bcb092445d16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39983 } }
I20260929 21:02:47.485178 15356 leader_election.cc:302] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bda88db6437c46fda958bcb092445d16; no voters:
I20260929 21:02:47.485816 15356 leader_election.cc:288] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 21:02:47.485951 15360 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 21:02:47.487174 15360 raft_consensus.cc:686] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [term 3 LEADER]: Becoming Leader. State: Replica: bda88db6437c46fda958bcb092445d16, State: Running, Role: LEADER
I20260929 21:02:47.487723 15360 consensus_queue.cc:237] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda88db6437c46fda958bcb092445d16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39983 } }
I20260929 21:02:47.487946 15356 sys_catalog.cc:564] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:02:47.490193 15362 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bda88db6437c46fda958bcb092445d16. Latest consensus state: current_term: 3 leader_uuid: "bda88db6437c46fda958bcb092445d16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda88db6437c46fda958bcb092445d16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39983 } } }
I20260929 21:02:47.490608 15362 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [sys.catalog]: This master's current role is: LEADER
I20260929 21:02:47.491202 15361 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "bda88db6437c46fda958bcb092445d16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda88db6437c46fda958bcb092445d16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39983 } } }
I20260929 21:02:47.491501 15361 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda88db6437c46fda958bcb092445d16 [sys.catalog]: This master's current role is: LEADER
I20260929 21:02:47.494042 15366 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:02:47.504029 15366 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:02:47.505666 15366 catalog_manager.cc:1259] Loaded cluster ID: 638bfd7a910c45d59fcd4bff57f4e24d
I20260929 21:02:47.505777 15366 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 21:02:47.508034 15366 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb85adcc9ad google::LogMessage::Fail() at ??:0
@ 0x7fb85adcf88a google::LogMessage::SendToLog() at ??:0
@ 0x7fb85adcc4de google::LogMessage::Flush() at ??:0
@ 0x7fb85adcda19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb8707ff512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fb8707fc976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb870884610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fb870884439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb85da18345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb85da1b91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fb85da1b799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb85d9f6eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb860cd06db start_thread at ??:0
@ 0x7fb859c6661f clone at ??:0
W20260929 21:02:47.827872 15358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:02:47.828183 15358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:02:47.828331 15358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:02:47.835570 15358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:02:47.835668 15358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:02:47.835778 15358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 21:02:47.845228 15358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:38137
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=42689
--tserver_master_addrs=127.0.97.62:39983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:02:47.846413 15358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:02:47.847602 15358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:02:47.860302 15395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:47.860332 15396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:47.860387 15398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:02:47.861052 15358 server_base.cc:1034] running on GCE node
I20260929 21:02:47.861619 15358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:02:47.862557 15358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:02:47.863765 15358 hybrid_clock.cc:648] HybridClock initialized: now 1790715767863717 us; error 39 us; skew 500 ppm
I20260929 21:02:47.864074 15358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:02:47 dist-test-slave-1dms krb5kdc[14211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:02:47.867302 15358 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 21:02:47.868717 15358 webserver.cc:460] Webserver started at http://127.0.97.1:42689/ using document root <none> and password file <none>
I20260929 21:02:47.869395 15358 fs_manager.cc:359] Metadata directory not provided
I20260929 21:02:47.869498 15358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:02:47.874568 15358 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260929 21:02:47.877280 15405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:47.878528 15358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 21:02:47.878696 15358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/wal
uuid: "fa23bbba7a9042d8a58b0e95f853c442"
format_stamp: "Formatted at 2026-09-29 21:02:42 on dist-test-slave-1dms"
server_key: "435963ed4503eab4e00650c9268868d8"
server_key_iv: "5becfc92daa06d74e00ba998bc25fc17"
server_key_version: "encryptionkey@0"
I20260929 21:02:47.879634 15358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:02:47.891849 15358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:02:47.895870 15358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:02:47.896147 15358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:02:47.897109 15358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:02:47.898824 15412 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260929 21:02:47.903867 15412 ts_tablet_manager.cc:1852] T d802e37f88c84d1e8a34b43ab087e0ca P fa23bbba7a9042d8a58b0e95f853c442: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260929 21:02:47.904821 15412 ts_tablet_manager.cc:1905] T d802e37f88c84d1e8a34b43ab087e0ca P fa23bbba7a9042d8a58b0e95f853c442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:02:47.911580 15412 ts_tablet_manager.cc:1918] T d802e37f88c84d1e8a34b43ab087e0ca P fa23bbba7a9042d8a58b0e95f853c442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:02:47.911773 15412 log.cc:1196] T d802e37f88c84d1e8a34b43ab087e0ca P fa23bbba7a9042d8a58b0e95f853c442: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/wal/wals/d802e37f88c84d1e8a34b43ab087e0ca
I20260929 21:02:47.912245 15412 ts_tablet_manager.cc:1939] T d802e37f88c84d1e8a34b43ab087e0ca P fa23bbba7a9042d8a58b0e95f853c442: Deleting consensus metadata
I20260929 21:02:47.912623 15358 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260929 21:02:47.912758 15358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.001s sys 0.000s
I20260929 21:02:47.912932 15358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:02:47.913025 15358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:47.946271 15358 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38137
I20260929 21:02:47.946333 15518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38137 every 8 connection(s)
I20260929 21:02:47.947659 15358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
I20260929 21:02:47.950207 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 15358
I20260929 21:02:47.950812 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:45349
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:39983
--webserver_port=43629
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 21:02:47.953132 15521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:47.950661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.1:49283)
0929 21:02:47.951642 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:47.951874 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":434,"threads_started":1}
W20260929 21:02:47.953822 15519 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39983 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:39983: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
I20260929 21:02:47.954821 15521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:47.954368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.1:38155)
0929 21:02:47.954563 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:47.954662 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 21:02:47.956272 15522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:47.955395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.1:45643)
0929 21:02:47.955972 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:47.956092 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":310,"threads_started":1}
W20260929 21:02:47.956552 15519 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 21:02:48.272738 15523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:02:48.273079 15523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:02:48.273176 15523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:02:48.280241 15523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:02:48.280352 15523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:02:48.280464 15523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 21:02:48.290239 15523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:45349
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=43629
--tserver_master_addrs=127.0.97.62:39983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:02:48.291477 15523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:02:48.292713 15523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:02:48.305655 15528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:48.305714 15531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:48.305711 15529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:02:48.306634 15523 server_base.cc:1034] running on GCE node
I20260929 21:02:48.307189 15523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:02:48.308071 15523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:02:48.309263 15523 hybrid_clock.cc:648] HybridClock initialized: now 1790715768309192 us; error 62 us; skew 500 ppm
I20260929 21:02:48.309535 15523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:02:48 dist-test-slave-1dms krb5kdc[14211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715768, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:02:48.312556 15523 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 21:02:48.313935 15523 webserver.cc:460] Webserver started at http://127.0.97.2:43629/ using document root <none> and password file <none>
I20260929 21:02:48.314558 15523 fs_manager.cc:359] Metadata directory not provided
I20260929 21:02:48.314654 15523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:02:48.319703 15523 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 21:02:48.324231 15538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:48.325592 15523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 21:02:48.325766 15523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/wal
uuid: "059743faeb2246d5b6a5510034eaa7cb"
format_stamp: "Formatted at 2026-09-29 21:02:42 on dist-test-slave-1dms"
server_key: "35491f0d67ca90a535ce21725f820bdc"
server_key_iv: "7818b9b569bcdb1f151e9d699149c3a9"
server_key_version: "encryptionkey@0"
I20260929 21:02:48.326732 15523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:02:48.361989 15523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:02:48.365842 15523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:02:48.366178 15523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:02:48.367151 15523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:02:48.368451 15523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:02:48.368561 15523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:48.368691 15523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:02:48.368767 15523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:48.401539 15523 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45349
I20260929 21:02:48.401629 15650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45349 every 8 connection(s)
I20260929 21:02:48.403017 15523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
I20260929 21:02:48.404124 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 15523
I20260929 21:02:48.404881 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:40965
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:39983
--webserver_port=39693
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 21:02:48.408442 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:48.406215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.2:44411)
0929 21:02:48.407052 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:48.407307 (+ 255us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":291,"threads_started":1}
W20260929 21:02:48.409159 15651 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39983 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:39983: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
I20260929 21:02:48.410043 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:48.409734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.2:36705)
0929 21:02:48.409877 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:48.409945 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 21:02:48.410923 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:48.410614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.2:49877)
0929 21:02:48.410741 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:48.410813 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 21:02:48.411182 15651 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 21:02:48.727033 15654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:02:48.727303 15654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:02:48.727401 15654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:02:48.734627 15654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:02:48.734740 15654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:02:48.734863 15654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 21:02:48.744307 15654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:40965
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=39693
--tserver_master_addrs=127.0.97.62:39983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:02:48.745536 15654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:02:48.746824 15654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:02:48.759588 15659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:48.759591 15660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:02:48.759672 15662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:02:48.760097 15654 server_base.cc:1034] running on GCE node
I20260929 21:02:48.760972 15654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:02:48.761890 15654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:02:48.763082 15654 hybrid_clock.cc:648] HybridClock initialized: now 1790715768763043 us; error 29 us; skew 500 ppm
I20260929 21:02:48.763401 15654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:02:48 dist-test-slave-1dms krb5kdc[14211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715768, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:02:48.766604 15654 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 21:02:48.768074 15654 webserver.cc:460] Webserver started at http://127.0.97.3:39693/ using document root <none> and password file <none>
I20260929 21:02:48.768755 15654 fs_manager.cc:359] Metadata directory not provided
I20260929 21:02:48.768914 15654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:02:48.773905 15654 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260929 21:02:48.776511 15669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:48.777840 15654 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 21:02:48.778013 15654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/wal
uuid: "c2dd5933f6464fa8a3ce2af782eacfb4"
format_stamp: "Formatted at 2026-09-29 21:02:43 on dist-test-slave-1dms"
server_key: "4b1b0d413af3cabbe47560980d78c6ad"
server_key_iv: "11ef40d280e0524898351c160bd39235"
server_key_version: "encryptionkey@0"
I20260929 21:02:48.778959 15654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:02:48.801442 15654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:02:48.805713 15654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:02:48.805976 15654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:02:48.806919 15654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:02:48.808255 15654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:02:48.808374 15654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:48.808511 15654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:02:48.808602 15654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:02:48.840508 15654 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40965
I20260929 21:02:48.840595 15781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40965 every 8 connection(s)
I20260929 21:02:48.841985 15654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
I20260929 21:02:48.846668 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:48.844897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.3:50229)
0929 21:02:48.845612 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:48.845820 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":270,"threads_started":1}
W20260929 21:02:48.847325 15782 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39983 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:39983: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
I20260929 21:02:48.848127 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 15654
I20260929 21:02:48.848176 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:48.847867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.3:58617)
0929 21:02:48.848008 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:48.848073 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 21:02:48.849068 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:48.848714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.3:36803)
0929 21:02:48.848888 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:48.848965 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 21:02:48.849284 15782 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 21:02:48.957803 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:48.957103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.1:40877)
0929 21:02:48.957514 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:48.957589 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":198,"threads_started":1}
I20260929 21:02:49.412590 15787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:49.411753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.2:34203)
0929 21:02:49.412274 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:49.412383 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":265,"threads_started":1}
I20260929 21:02:49.850428 15788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:49.849713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.3:35949)
0929 21:02:49.850131 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:49.850210 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":204,"threads_started":1}
I20260929 21:02:49.959407 15789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:49.958756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.1:37531)
0929 21:02:49.959177 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:49.959253 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":226,"threads_started":1}
I20260929 21:02:50.414129 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:50.413457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.2:55649)
0929 21:02:50.413891 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:50.413969 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":215,"threads_started":1}
I20260929 21:02:50.851951 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:50.851288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.3:55461)
0929 21:02:50.851715 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:50.851798 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":204,"threads_started":1}
I20260929 21:02:50.961100 15792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:50.960350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.1:49843)
0929 21:02:50.960785 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:50.960919 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":225,"threads_started":1}
I20260929 21:02:51.415860 15793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:51.415073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.2:59447)
0929 21:02:51.415575 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:51.415678 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":260,"threads_started":1}
I20260929 21:02:51.853493 15794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:51.852776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.3:56803)
0929 21:02:51.853250 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:51.853334 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":274,"threads_started":1}
I20260929 21:02:51.962728 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:51.962020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.1:57749)
0929 21:02:51.962464 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:51.962557 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":233,"threads_started":1}
I20260929 21:02:52.417578 15796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:52.416788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.2:50971)
0929 21:02:52.417346 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:52.417418 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":277,"threads_started":1}
I20260929 21:02:52.854938 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:52.854265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.3:56863)
0929 21:02:52.854672 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:52.854746 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":210,"threads_started":1}
I20260929 21:02:52.964290 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:52.963593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.1:36105)
0929 21:02:52.964037 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:52.964116 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":237,"threads_started":1}
I20260929 21:02:53.419116 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:53.418453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.2:52835)
0929 21:02:53.418883 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:53.418955 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":235,"threads_started":1}
I20260929 21:02:53.856702 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:53.855793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.3:56545)
0929 21:02:53.856337 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:53.856452 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":313,"threads_started":1}
I20260929 21:02:53.966095 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:53.965285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.1:41859)
0929 21:02:53.965825 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:53.965922 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":294,"threads_started":1}
I20260929 21:02:54.420692 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:54.420016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.2:55403)
0929 21:02:54.420452 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:54.420537 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":235,"threads_started":1}
I20260929 21:02:54.858253 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:54.857549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.3:41751)
0929 21:02:54.857964 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:54.858094 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":220,"threads_started":1}
I20260929 21:02:54.967751 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:54.966956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.1:60281)
0929 21:02:54.967452 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:54.967532 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":232,"threads_started":1}
I20260929 21:02:55.422371 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:55.421626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.2:54695)
0929 21:02:55.422077 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:55.422161 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":196,"threads_started":1}
I20260929 21:02:55.859802 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:55.859088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.3:53711)
0929 21:02:55.859530 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:55.859603 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":203,"threads_started":1}
I20260929 21:02:55.969350 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:55.968651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.1:37715)
0929 21:02:55.969098 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:55.969179 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":249,"threads_started":1}
I20260929 21:02:56.423831 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:56.423180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.2:46481)
0929 21:02:56.423609 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:56.423682 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":233,"threads_started":1}
I20260929 21:02:56.861418 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:56.860657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.3:58449)
0929 21:02:56.861177 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:56.861257 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":310,"threads_started":1}
I20260929 21:02:56.970867 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:56.970158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.1:45879)
0929 21:02:56.970604 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:56.970680 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":231,"threads_started":1}
I20260929 21:02:57.425522 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:57.424726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.2:49705)
0929 21:02:57.425265 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:57.425353 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":279,"threads_started":1}
I20260929 21:02:57.862941 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:57.862213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.3:39725)
0929 21:02:57.862683 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:57.862764 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":240,"threads_started":1}
I20260929 21:02:57.972409 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:57.971748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.1:32779)
0929 21:02:57.972170 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:57.972247 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":209,"threads_started":1}
I20260929 21:02:58.427279 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:58.426597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.2:52819)
0929 21:02:58.427029 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:58.427122 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":230,"threads_started":1}
I20260929 21:02:58.864485 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:58.863794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.3:48359)
0929 21:02:58.864205 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:58.864279 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":225,"threads_started":1}
I20260929 21:02:58.974023 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:58.973334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.1:40283)
0929 21:02:58.973777 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:58.973864 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":239,"threads_started":1}
I20260929 21:02:59.428815 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:59.428127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.2:41995)
0929 21:02:59.428557 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:59.428643 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":220,"threads_started":1}
I20260929 21:02:59.866000 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:59.865330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.3:59637)
0929 21:02:59.865734 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:59.865818 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":206,"threads_started":1}
I20260929 21:02:59.975786 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:02:59.974888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.1:36219)
0929 21:02:59.975423 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0929 21:02:59.975535 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":233,"threads_started":1}
I20260929 21:03:00.430480 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:00.429810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.2:38179)
0929 21:03:00.430236 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:00.430312 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":209,"threads_started":1}
I20260929 21:03:00.867669 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:00.866935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.3:45767)
0929 21:03:00.867422 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:00.867513 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":280,"threads_started":1}
I20260929 21:03:00.977547 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:00.976797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.1:56487)
0929 21:03:00.977290 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:00.977381 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":221,"threads_started":1}
I20260929 21:03:01.432159 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:01.431411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.2:48139)
0929 21:03:01.431898 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:01.431978 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":258,"threads_started":1}
I20260929 21:03:01.869292 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:01.868520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.3:58591)
0929 21:03:01.869033 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:01.869112 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":216,"threads_started":1}
I20260929 21:03:01.979271 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:01.978479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.1:54001)
0929 21:03:01.978950 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:01.979029 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":246,"threads_started":1}
I20260929 21:03:02.433728 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:02.433070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.2:33997)
0929 21:03:02.433483 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:02.433558 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":201,"threads_started":1}
I20260929 21:03:02.870829 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:02.870158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.3:57589)
0929 21:03:02.870573 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:02.870652 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":214,"threads_started":1}
I20260929 21:03:02.981168 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:02.980367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.1:58511)
0929 21:03:02.980885 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:02.980980 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":241,"threads_started":1}
I20260929 21:03:03.435464 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:03.434772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.2:52935)
0929 21:03:03.435205 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:03.435279 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":234,"threads_started":1}
I20260929 21:03:03.861398 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 15358
I20260929 21:03:03.873037 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:03.871963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39983 (local address 127.0.97.3:38161)
0929 21:03:03.872664 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:03.872763 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":472,"threads_started":1}
I20260929 21:03:03.885435 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:03.862645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33128 (local address 127.0.97.1:38137)
0929 21:03:03.863236 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0929 21:03:03.863250 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 21:03:03.863288 (+ 38us) server_negotiation.cc:407] Connection header received
0929 21:03:03.864991 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:03:03.865007 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:03:03.865703 (+ 696us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:03:03.866481 (+ 778us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:03:03.868588 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:03.870483 (+ 1895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:03.872127 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:03.873356 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:03.873589 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:03.876935 (+ 3346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:03:03.876944 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:03:03.876960 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:03:03.876998 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:03:03.880608 (+ 3610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:03:03.881823 (+ 1215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:03:03.881832 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:03:03.881848 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:03:03.882136 (+ 288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:03:03.882712 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:03:03.882718 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:03:03.882721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:03:03.883606 (+ 885us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:03:03.883908 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 21:03:03.884419 (+ 511us) server_negotiation.cc:299] Negotiation successful
0929 21:03:03.884916 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":325,"threads_started":1}
I20260929 21:03:03.887195 15514 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:03:03.984431 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 15358
I20260929 21:03:03.998162 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 15523
I20260929 21:03:04.014623 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:03.999044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49232 (local address 127.0.97.2:45349)
0929 21:03:03.999616 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0929 21:03:03.999629 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 21:03:03.999666 (+ 37us) server_negotiation.cc:407] Connection header received
0929 21:03:04.001258 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:03:04.001268 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:03:04.001694 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:03:04.001994 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:03:04.003211 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:04.004398 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:04.005747 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:04.006415 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:04.006544 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:04.009068 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:03:04.009075 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:03:04.009089 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:03:04.009124 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:03:04.011784 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:03:04.012393 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:03:04.012399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:03:04.012405 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:03:04.012557 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:03:04.013006 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:03:04.013009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:03:04.013011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:03:04.013486 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:03:04.013719 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 21:03:04.014047 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 21:03:04.014348 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":294,"threads_started":1}
I20260929 21:03:04.015955 15646 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:03:04.100571 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 15523
I20260929 21:03:04.118080 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 15654
I20260929 21:03:04.134786 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:04.119155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59028 (local address 127.0.97.3:40965)
0929 21:03:04.119758 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0929 21:03:04.119773 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 21:03:04.119814 (+ 41us) server_negotiation.cc:407] Connection header received
0929 21:03:04.121502 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:03:04.121512 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:03:04.121930 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:03:04.122217 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:03:04.123466 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:04.124622 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:04.125844 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:04.126507 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:04.126660 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:04.129154 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:03:04.129161 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:03:04.129176 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:03:04.129210 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:03:04.131905 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:03:04.132571 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:03:04.132577 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:03:04.132583 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:03:04.132735 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:03:04.133107 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:03:04.133110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:03:04.133112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:03:04.133607 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:03:04.133830 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 21:03:04.134177 (+ 347us) server_negotiation.cc:299] Negotiation successful
0929 21:03:04.134500 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":306,"threads_started":1}
I20260929 21:03:04.136148 15777 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:03:04.220281 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 15654
2026-09-29T21:03:04Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24127 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 21:03:04 dist-test-slave-1dms krb5kdc[15840](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 21:03:04 dist-test-slave-1dms krb5kdc[15840](info): set up 2 sockets
Sep 29 21:03:04 dist-test-slave-1dms krb5kdc[15840](info): commencing operation
krb5kdc: starting...
W20260929 21:03:05.321360 388 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 21:03:05 dist-test-slave-1dms krb5kdc[15840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T21:03:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:03:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 21:03:05.555157 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46807
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:03:05.879637 15853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:03:05.879920 15853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:03:05.880009 15853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:03:05.887101 15853 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 21:03:05.887197 15853 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 21:03:05.887233 15853 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 21:03:05.887284 15853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:03:05.887341 15853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:03:05.887405 15853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:03:05.887450 15853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:03:05.887496 15853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:03:05.897049 15853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:03:05.898301 15853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:03:05.899554 15853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:03:05.911314 15858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:03:05.911267 15859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:03:05.911389 15861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:03:05.912209 15853 server_base.cc:1034] running on GCE node
I20260929 21:03:05.913098 15853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:03:05.914459 15853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:03:05.915730 15853 hybrid_clock.cc:648] HybridClock initialized: now 1790715785915674 us; error 56 us; skew 500 ppm
I20260929 21:03:05.916055 15853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:03:05 dist-test-slave-1dms krb5kdc[15840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:03:05.919268 15853 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 21:03:05.920682 15853 webserver.cc:460] Webserver started at http://127.0.97.62:43369/ using document root <none> and password file <none>
I20260929 21:03:05.921416 15853 fs_manager.cc:359] Metadata directory not provided
I20260929 21:03:05.921545 15853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:03:05.921857 15853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:03:05.924764 15853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "885b912376d24bc48fda185253979e21"
format_stamp: "Formatted at 2026-09-29 21:03:05 on dist-test-slave-1dms"
server_key: "516751997f5064282644b29234ab7c4e"
server_key_iv: "649df24baeee8faf33fbf6fc42ba08b4"
server_key_version: "encryptionkey@0"
I20260929 21:03:05.925622 15853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "885b912376d24bc48fda185253979e21"
format_stamp: "Formatted at 2026-09-29 21:03:05 on dist-test-slave-1dms"
server_key: "516751997f5064282644b29234ab7c4e"
server_key_iv: "649df24baeee8faf33fbf6fc42ba08b4"
server_key_version: "encryptionkey@0"
I20260929 21:03:05.930789 15853 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 21:03:05.934258 15868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:03:05.935503 15853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 21:03:05.935684 15853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "885b912376d24bc48fda185253979e21"
format_stamp: "Formatted at 2026-09-29 21:03:05 on dist-test-slave-1dms"
server_key: "516751997f5064282644b29234ab7c4e"
server_key_iv: "649df24baeee8faf33fbf6fc42ba08b4"
server_key_version: "encryptionkey@0"
I20260929 21:03:05.935897 15853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:03:05.965044 15853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:03:05.969175 15853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:03:05.969458 15853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:03:05.988425 15853 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46807
I20260929 21:03:05.988448 15919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46807 every 8 connection(s)
I20260929 21:03:05.989812 15853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:03:05.993521 15920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:03:05.999375 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 15853
I20260929 21:03:05.999599 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:03:06.000406 388 external_mini_cluster.cc:1428] Setting key 7b4d7bb3557a4e020c6e98b81e815664
I20260929 21:03:06.004689 15920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21: Bootstrap starting.
I20260929 21:03:06.008494 15920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21: Neither blocks nor log segments found. Creating new log.
I20260929 21:03:06.009681 15920 log.cc:824] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21: Log is configured to *not* fsync() on all Append() calls
I20260929 21:03:06.012677 15920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21: No bootstrap required, opened a new log
Sep 29 21:03:06 dist-test-slave-1dms krb5kdc[15840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:03:06.023337 15920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885b912376d24bc48fda185253979e21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46807 } }
I20260929 21:03:06.023763 15920 raft_consensus.cc:374] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:03:06.023873 15920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 885b912376d24bc48fda185253979e21, State: Initialized, Role: FOLLOWER
I20260929 21:03:06.024531 15920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885b912376d24bc48fda185253979e21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46807 } }
I20260929 21:03:06.024819 15920 raft_consensus.cc:388] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:03:06.024981 15920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:03:06.025171 15923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:06.002763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34862 (local address 127.0.97.62:46807)
0929 21:03:06.003547 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0929 21:03:06.003558 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 21:03:06.003600 (+ 42us) server_negotiation.cc:407] Connection header received
0929 21:03:06.005346 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:03:06.005375 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:03:06.005745 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:03:06.006086 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:03:06.007962 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:06.009416 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:06.011182 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:06.011829 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:06.011961 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:06.016716 (+ 4755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:03:06.016731 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:03:06.016750 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:03:06.016809 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:03:06.020823 (+ 4014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:03:06.021502 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:03:06.021511 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:03:06.021524 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:03:06.021748 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:03:06.022071 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:03:06.022076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:03:06.022079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:03:06.022758 (+ 679us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:03:06.023045 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0929 21:03:06.023342 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 21:03:06.023780 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":366,"threads_started":1}
I20260929 21:03:06.025175 15920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:03:06.027525 15920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885b912376d24bc48fda185253979e21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46807 } }
I20260929 21:03:06.028013 15920 leader_election.cc:302] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 885b912376d24bc48fda185253979e21; no voters:
I20260929 21:03:06.028570 15920 leader_election.cc:288] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 21:03:06.028697 15925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:03:06.029430 15925 raft_consensus.cc:686] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [term 1 LEADER]: Becoming Leader. State: Replica: 885b912376d24bc48fda185253979e21, State: Running, Role: LEADER
I20260929 21:03:06.029886 15925 consensus_queue.cc:237] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885b912376d24bc48fda185253979e21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46807 } }
I20260929 21:03:06.030653 15920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:03:06.033030 15926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "885b912376d24bc48fda185253979e21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885b912376d24bc48fda185253979e21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46807 } } }
I20260929 21:03:06.033002 15927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 885b912376d24bc48fda185253979e21. Latest consensus state: current_term: 1 leader_uuid: "885b912376d24bc48fda185253979e21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885b912376d24bc48fda185253979e21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46807 } } }
I20260929 21:03:06.033459 15926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [sys.catalog]: This master's current role is: LEADER
I20260929 21:03:06.033504 15927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [sys.catalog]: This master's current role is: LEADER
I20260929 21:03:06.037659 15932 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:03:06.042537 15932 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:03:06.052410 15932 catalog_manager.cc:1347] Generated new cluster ID: 0d536ebc009246f2ae6910666b207a89
I20260929 21:03:06.052520 15932 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:03:06.070904 15932 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:03:06.072192 15932 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:03:06.086445 15932 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21: Generated new TSK 0
I20260929 21:03:06.087265 15932 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 21:03:06.144209 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46807
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:03:06.470060 15946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:03:06.470372 15946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:03:06.470539 15946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:03:06.477721 15946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:03:06.477859 15946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:03:06.478039 15946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 21:03:06.487557 15946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:03:06.488819 15946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:03:06.490123 15946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:03:06.504030 15952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:03:06.504050 15954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:03:06.504637 15946 server_base.cc:1034] running on GCE node
W20260929 21:03:06.505481 15951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:03:06.506313 15946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:03:06.507546 15946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:03:06.508857 15946 hybrid_clock.cc:648] HybridClock initialized: now 1790715786508777 us; error 46 us; skew 500 ppm
I20260929 21:03:06.509317 15946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:03:06 dist-test-slave-1dms krb5kdc[15840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:03:06.513145 15946 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 21:03:06.514484 15946 webserver.cc:460] Webserver started at http://127.0.97.1:33381/ using document root <none> and password file <none>
I20260929 21:03:06.515153 15946 fs_manager.cc:359] Metadata directory not provided
I20260929 21:03:06.515295 15946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:03:06.515600 15946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:03:06.518469 15946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/data/instance:
uuid: "746b2763ccbf45d5874eb98263f37a3f"
format_stamp: "Formatted at 2026-09-29 21:03:06 on dist-test-slave-1dms"
server_key: "34cd18bd130d4157d17121a84ce76ee5"
server_key_iv: "cf2f5f75793e972df16db7d45331bc91"
server_key_version: "encryptionkey@0"
I20260929 21:03:06.519198 15946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/wal/instance:
uuid: "746b2763ccbf45d5874eb98263f37a3f"
format_stamp: "Formatted at 2026-09-29 21:03:06 on dist-test-slave-1dms"
server_key: "34cd18bd130d4157d17121a84ce76ee5"
server_key_iv: "cf2f5f75793e972df16db7d45331bc91"
server_key_version: "encryptionkey@0"
I20260929 21:03:06.524454 15946 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260929 21:03:06.528012 15961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:03:06.529367 15946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:03:06.529533 15946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/wal
uuid: "746b2763ccbf45d5874eb98263f37a3f"
format_stamp: "Formatted at 2026-09-29 21:03:06 on dist-test-slave-1dms"
server_key: "34cd18bd130d4157d17121a84ce76ee5"
server_key_iv: "cf2f5f75793e972df16db7d45331bc91"
server_key_version: "encryptionkey@0"
I20260929 21:03:06.529721 15946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:03:06.552018 15946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:03:06.556150 15946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:03:06.556449 15946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:03:06.557511 15946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:03:06.558904 15946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:03:06.558976 15946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:03:06.559119 15946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:03:06.559183 15946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:03:06.592983 15946 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46765
I20260929 21:03:06.593070 16073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46765 every 8 connection(s)
I20260929 21:03:06.594254 15946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
I20260929 21:03:06.598883 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 15946
I20260929 21:03:06.599107 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/wal/instance
I20260929 21:03:06.599936 388 external_mini_cluster.cc:1428] Setting key 1ee7329739276b7dfb5b0b8266cd44cf
Sep 29 21:03:06 dist-test-slave-1dms krb5kdc[15840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:03:06.617739 16076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:06.596767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59229 (local address 127.0.97.62:46807)
0929 21:03:06.597315 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0929 21:03:06.597320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:03:06.598148 (+ 828us) server_negotiation.cc:407] Connection header received
0929 21:03:06.600757 (+ 2609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:03:06.600763 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:03:06.600912 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:03:06.601030 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:03:06.602749 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:06.603610 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:06.606053 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:06.606464 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:06.606568 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:06.610793 (+ 4225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:03:06.610802 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:03:06.610808 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:03:06.610851 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:03:06.614514 (+ 3663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:03:06.615222 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:03:06.615230 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:03:06.615236 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:03:06.615394 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:03:06.615786 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:03:06.615793 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:03:06.615799 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:03:06.616172 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:03:06.616340 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 21:03:06.616997 (+ 657us) server_negotiation.cc:299] Negotiation successful
0929 21:03:06.617248 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":305,"threads_started":1}
I20260929 21:03:06.618338 16077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:06.597262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.1:59229)
0929 21:03:06.597990 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:06.598027 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 21:03:06.600460 (+ 2433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:03:06.601266 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:03:06.601274 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:03:06.601774 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:03:06.602571 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:03:06.602598 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:03:06.603773 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:03:06.603782 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:03:06.605920 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:03:06.605937 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:03:06.606695 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:03:06.606705 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:03:06.607054 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:06.607800 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:03:06.607838 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:03:06.610566 (+ 2728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:03:06.614651 (+ 4085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:03:06.614657 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:03:06.614669 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:03:06.615084 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:03:06.615496 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:03:06.615499 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:03:06.615502 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:03:06.615674 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:03:06.616287 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:03:06.616292 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:03:06.616602 (+ 310us) client_negotiation.cc:769] Sending connection context
0929 21:03:06.616818 (+ 216us) client_negotiation.cc:240] Negotiation successful
0929 21:03:06.617253 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":292,"threads_started":1}
I20260929 21:03:06.619884 16074 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46807
I20260929 21:03:06.620187 16074 heartbeater.cc:464] Registering TS with master...
I20260929 21:03:06.620966 16074 heartbeater.cc:511] Master 127.0.97.62:46807 requested a full tablet report, sending...
I20260929 21:03:06.622956 15885 ts_manager.cc:194] Registered new tserver with Master: 746b2763ccbf45d5874eb98263f37a3f (127.0.97.1:46765)
I20260929 21:03:06.624308 15885 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59229
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 21:03:06.656131 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46807
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:03:06.976822 16080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:03:06.977151 16080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:03:06.977335 16080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:03:06.984539 16080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:03:06.984675 16080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:03:06.984884 16080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 21:03:06.994583 16080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:03:06.995822 16080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:03:06.997159 16080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:03:07.009891 16085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:03:07.010342 16086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:03:07.010389 16088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:03:07.010983 16080 server_base.cc:1034] running on GCE node
I20260929 21:03:07.011535 16080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:03:07.012427 16080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:03:07.013618 16080 hybrid_clock.cc:648] HybridClock initialized: now 1790715787013578 us; error 32 us; skew 500 ppm
I20260929 21:03:07.013893 16080 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:03:07 dist-test-slave-1dms krb5kdc[15840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:03:07.017023 16080 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 21:03:07.018478 16080 webserver.cc:460] Webserver started at http://127.0.97.2:37831/ using document root <none> and password file <none>
I20260929 21:03:07.019136 16080 fs_manager.cc:359] Metadata directory not provided
I20260929 21:03:07.019240 16080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:03:07.019510 16080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:03:07.022413 16080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/data/instance:
uuid: "e5726fa404854fc4ba03654f587816d8"
format_stamp: "Formatted at 2026-09-29 21:03:07 on dist-test-slave-1dms"
server_key: "7cad1c146ac263cca6b5a42bf890ae28"
server_key_iv: "42dc0d4024cce989637d4b869183890f"
server_key_version: "encryptionkey@0"
I20260929 21:03:07.023078 16080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e5726fa404854fc4ba03654f587816d8"
format_stamp: "Formatted at 2026-09-29 21:03:07 on dist-test-slave-1dms"
server_key: "7cad1c146ac263cca6b5a42bf890ae28"
server_key_iv: "42dc0d4024cce989637d4b869183890f"
server_key_version: "encryptionkey@0"
I20260929 21:03:07.028067 16080 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 21:03:07.031569 16095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:03:07.033066 16080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:03:07.033190 16080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/wal
uuid: "e5726fa404854fc4ba03654f587816d8"
format_stamp: "Formatted at 2026-09-29 21:03:07 on dist-test-slave-1dms"
server_key: "7cad1c146ac263cca6b5a42bf890ae28"
server_key_iv: "42dc0d4024cce989637d4b869183890f"
server_key_version: "encryptionkey@0"
I20260929 21:03:07.033352 16080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:03:07.066222 16080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:03:07.070057 16080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:03:07.070290 16080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:03:07.071151 16080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:03:07.072360 16080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:03:07.072428 16080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:03:07.072515 16080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:03:07.072556 16080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:03:07.105652 16080 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33721
I20260929 21:03:07.105697 16207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33721 every 8 connection(s)
I20260929 21:03:07.106968 16080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
I20260929 21:03:07.109658 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 16080
I20260929 21:03:07.109865 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/wal/instance
I20260929 21:03:07.110359 388 external_mini_cluster.cc:1428] Setting key 5687363e40e849e68c9f8e01d2ba8402
Sep 29 21:03:07 dist-test-slave-1dms krb5kdc[15840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:03:07.129053 16076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:07.109774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45471 (local address 127.0.97.62:46807)
0929 21:03:07.109986 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 21:03:07.109992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:03:07.111281 (+ 1289us) server_negotiation.cc:407] Connection header received
0929 21:03:07.113666 (+ 2385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:03:07.113678 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:03:07.113742 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:03:07.113862 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:03:07.115977 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:07.116669 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:07.119216 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:07.119633 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:07.119729 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:07.123376 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:03:07.123381 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:03:07.123384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:03:07.123421 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:03:07.126046 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:03:07.126775 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:03:07.126787 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:03:07.126790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:03:07.126910 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:03:07.127350 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:03:07.127355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:03:07.127358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:03:07.127619 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:03:07.127798 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 21:03:07.128454 (+ 656us) server_negotiation.cc:299] Negotiation successful
0929 21:03:07.128676 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 21:03:07.130009 16210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:07.110296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.2:45471)
0929 21:03:07.111109 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:07.111147 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 21:03:07.113372 (+ 2225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:03:07.114129 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:03:07.114143 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:03:07.114968 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:03:07.115784 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:03:07.115813 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:03:07.116826 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:03:07.116859 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:03:07.119069 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:03:07.119087 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:03:07.119861 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:03:07.119870 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:03:07.120203 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:07.121003 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:03:07.121043 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:03:07.123144 (+ 2101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:03:07.126198 (+ 3054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:03:07.126208 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:03:07.126225 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:03:07.126634 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:03:07.127032 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:03:07.127037 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:03:07.127041 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:03:07.127231 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:03:07.127751 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:03:07.127761 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:03:07.128092 (+ 331us) client_negotiation.cc:769] Sending connection context
0929 21:03:07.128355 (+ 263us) client_negotiation.cc:240] Negotiation successful
0929 21:03:07.128904 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":329,"threads_started":1}
I20260929 21:03:07.131524 16208 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46807
I20260929 21:03:07.131821 16208 heartbeater.cc:464] Registering TS with master...
I20260929 21:03:07.132630 16208 heartbeater.cc:511] Master 127.0.97.62:46807 requested a full tablet report, sending...
I20260929 21:03:07.134303 15885 ts_manager.cc:194] Registered new tserver with Master: e5726fa404854fc4ba03654f587816d8 (127.0.97.2:33721)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 21:03:07.135141 15885 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45471
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 21:03:07.163717 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46807
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:03:07.481048 16213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:03:07.481381 16213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:03:07.481549 16213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:03:07.488653 16213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:03:07.488809 16213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:03:07.489001 16213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 21:03:07.498454 16213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:03:07.499723 16213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:03:07.501077 16213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:03:07.514276 16218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:03:07.514317 16221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:03:07.514276 16219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:03:07.515136 16213 server_base.cc:1034] running on GCE node
I20260929 21:03:07.515762 16213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:03:07.516587 16213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:03:07.517781 16213 hybrid_clock.cc:648] HybridClock initialized: now 1790715787517705 us; error 62 us; skew 500 ppm
I20260929 21:03:07.518100 16213 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:03:07 dist-test-slave-1dms krb5kdc[15840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:03:07.521042 16213 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 21:03:07.522346 16213 webserver.cc:460] Webserver started at http://127.0.97.3:36963/ using document root <none> and password file <none>
I20260929 21:03:07.522997 16213 fs_manager.cc:359] Metadata directory not provided
I20260929 21:03:07.523097 16213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:03:07.523335 16213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:03:07.526136 16213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/data/instance:
uuid: "3f66c38d69d34c9194b4d1264eddeb3d"
format_stamp: "Formatted at 2026-09-29 21:03:07 on dist-test-slave-1dms"
server_key: "b0facff35d5cf4a30c4a791c4c936e1d"
server_key_iv: "3944b588241ddd381526015d31495ff4"
server_key_version: "encryptionkey@0"
I20260929 21:03:07.526813 16213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/wal/instance:
uuid: "3f66c38d69d34c9194b4d1264eddeb3d"
format_stamp: "Formatted at 2026-09-29 21:03:07 on dist-test-slave-1dms"
server_key: "b0facff35d5cf4a30c4a791c4c936e1d"
server_key_iv: "3944b588241ddd381526015d31495ff4"
server_key_version: "encryptionkey@0"
I20260929 21:03:07.531654 16213 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 21:03:07.535171 16228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:03:07.536410 16213 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 21:03:07.536584 16213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/wal
uuid: "3f66c38d69d34c9194b4d1264eddeb3d"
format_stamp: "Formatted at 2026-09-29 21:03:07 on dist-test-slave-1dms"
server_key: "b0facff35d5cf4a30c4a791c4c936e1d"
server_key_iv: "3944b588241ddd381526015d31495ff4"
server_key_version: "encryptionkey@0"
I20260929 21:03:07.536773 16213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:03:07.566524 16213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:03:07.570833 16213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:03:07.571141 16213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:03:07.572156 16213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:03:07.573561 16213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:03:07.573634 16213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:03:07.573764 16213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:03:07.573825 16213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:03:07.607175 16213 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36811
I20260929 21:03:07.607219 16340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36811 every 8 connection(s)
I20260929 21:03:07.608441 16213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
I20260929 21:03:07.615535 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 16213
I20260929 21:03:07.615726 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/wal/instance
I20260929 21:03:07.616173 388 external_mini_cluster.cc:1428] Setting key 9ad0e5d97776de892660533666b94437
Sep 29 21:03:07 dist-test-slave-1dms krb5kdc[15840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:03:07.627260 16074 heartbeater.cc:503] Master 127.0.97.62:46807 was elected leader, sending a full tablet report...
I20260929 21:03:07.628544 16076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:07.610941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45349 (local address 127.0.97.62:46807)
0929 21:03:07.611107 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 21:03:07.611111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:03:07.612144 (+ 1033us) server_negotiation.cc:407] Connection header received
0929 21:03:07.613953 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:03:07.613960 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:03:07.614077 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:03:07.614187 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:03:07.615924 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:07.616870 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:07.618866 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:07.619376 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:07.619472 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:07.622872 (+ 3400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:03:07.622876 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:03:07.622879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:03:07.622909 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:03:07.625508 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:03:07.626219 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:03:07.626225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:03:07.626229 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:03:07.626362 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:03:07.626774 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:03:07.626781 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:03:07.626784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:03:07.627058 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:03:07.627212 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 21:03:07.627971 (+ 759us) server_negotiation.cc:299] Negotiation successful
0929 21:03:07.628183 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 21:03:07.629524 16343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:07.611329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.3:45349)
0929 21:03:07.611987 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:07.612021 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 21:03:07.613681 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:03:07.614405 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:03:07.614413 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:03:07.614920 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:03:07.615702 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:03:07.615729 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:03:07.617030 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:03:07.617035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:03:07.618720 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:03:07.618732 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:03:07.619555 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:03:07.619560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:03:07.619742 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:07.620399 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:03:07.620420 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:03:07.622608 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:03:07.625655 (+ 3047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:03:07.625663 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:03:07.625679 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:03:07.626081 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:03:07.626470 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:03:07.626476 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:03:07.626479 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:03:07.626665 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:03:07.627185 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:03:07.627193 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:03:07.627585 (+ 392us) client_negotiation.cc:769] Sending connection context
0929 21:03:07.627837 (+ 252us) client_negotiation.cc:240] Negotiation successful
0929 21:03:07.628347 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":263,"threads_started":1}
I20260929 21:03:07.630918 16341 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46807
I20260929 21:03:07.631243 16341 heartbeater.cc:464] Registering TS with master...
I20260929 21:03:07.632038 16341 heartbeater.cc:511] Master 127.0.97.62:46807 requested a full tablet report, sending...
I20260929 21:03:07.633512 15885 ts_manager.cc:194] Registered new tserver with Master: 3f66c38d69d34c9194b4d1264eddeb3d (127.0.97.3:36811)
I20260929 21:03:07.634254 15885 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45349
I20260929 21:03:07.643406 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 21:03:07.659327 16076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:07.647599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34866 (local address 127.0.97.62:46807)
0929 21:03:07.647757 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 21:03:07.647761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:03:07.648244 (+ 483us) server_negotiation.cc:407] Connection header received
0929 21:03:07.648376 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:03:07.648379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:03:07.648435 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:03:07.648516 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:03:07.650026 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:07.650711 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:07.652035 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:07.652416 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:07.652501 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:07.654812 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:03:07.654817 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:03:07.654819 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:03:07.654848 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:03:07.657179 (+ 2331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:03:07.657810 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:03:07.657814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:03:07.657816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:03:07.657921 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:03:07.658208 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:03:07.658210 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:03:07.658212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:03:07.658503 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:03:07.658680 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 21:03:07.658945 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 21:03:07.659093 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 21:03:07.663004 15885 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34866:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 21:03:07.669668 15885 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 21:03:07.698289 16354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:07.680931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46765 (local address 127.0.0.1:60726)
0929 21:03:07.688944 (+ 8013us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:07.688997 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 21:03:07.689160 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:03:07.690267 (+ 1107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:03:07.690278 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:03:07.690333 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:03:07.690935 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:03:07.690963 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:03:07.693278 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:03:07.693283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:03:07.695455 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:03:07.695466 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:03:07.697235 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:03:07.697241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:03:07.697423 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:07.697469 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 21:03:07.697601 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 21:03:07.697790 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7730,"mutex_wait_us":46,"spinlock_wait_cycles":2944,"thread_start_us":7702,"threads_started":1}
I20260929 21:03:07.698836 16358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:07.689576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49620 (local address 127.0.97.2:33721)
0929 21:03:07.691450 (+ 1874us) server_negotiation.cc:206] Beginning negotiation
0929 21:03:07.691458 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:03:07.691482 (+ 24us) server_negotiation.cc:407] Connection header received
0929 21:03:07.691567 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:03:07.691576 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:03:07.691798 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:03:07.691995 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:03:07.692532 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:07.693800 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:07.696209 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:07.697222 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:07.697352 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:07.697542 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 21:03:07.698388 (+ 846us) server_negotiation.cc:299] Negotiation successful
0929 21:03:07.698550 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1730,"thread_start_us":1630,"threads_started":1}
I20260929 21:03:07.698818 16356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:07.681713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41816 (local address 127.0.97.3:36811)
0929 21:03:07.688384 (+ 6671us) server_negotiation.cc:206] Beginning negotiation
0929 21:03:07.688397 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 21:03:07.689089 (+ 692us) server_negotiation.cc:407] Connection header received
0929 21:03:07.689417 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:03:07.689427 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:03:07.689706 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:03:07.689914 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:03:07.691723 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:07.693066 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:07.695976 (+ 2910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:07.697380 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:07.697505 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:07.697729 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 21:03:07.698317 (+ 588us) server_negotiation.cc:299] Negotiation successful
0929 21:03:07.698486 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6458,"thread_start_us":6293,"threads_started":1}
I20260929 21:03:07.699120 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:07.681181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60726 (local address 127.0.97.1:46765)
0929 21:03:07.687974 (+ 6793us) server_negotiation.cc:206] Beginning negotiation
0929 21:03:07.687987 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 21:03:07.689198 (+ 1211us) server_negotiation.cc:407] Connection header received
0929 21:03:07.689434 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:03:07.689445 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:03:07.689687 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:03:07.689902 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:03:07.691636 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:07.693174 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:07.695603 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:07.697025 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:07.697148 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:07.698470 (+ 1322us) server_negotiation.cc:1035] Waiting for connection context
0929 21:03:07.698530 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 21:03:07.698687 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6556,"thread_start_us":6364,"threads_started":1}
I20260929 21:03:07.699437 16357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:07.689388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33721 (local address 127.0.0.1:49620)
0929 21:03:07.690878 (+ 1490us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:07.690896 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 21:03:07.691014 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:03:07.692003 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:03:07.692008 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:03:07.692035 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:03:07.692376 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:03:07.692387 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:03:07.693998 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:03:07.694005 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:03:07.696072 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:03:07.696081 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:03:07.697686 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:03:07.697692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:03:07.697867 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:07.697958 (+ 91us) client_negotiation.cc:769] Sending connection context
0929 21:03:07.698046 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 21:03:07.698165 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1344,"mutex_wait_us":42,"thread_start_us":1203,"threads_started":1}
I20260929 21:03:07.699844 16353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:07.681031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36811 (local address 127.0.0.1:41816)
0929 21:03:07.688944 (+ 7913us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:07.688994 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 21:03:07.689194 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:03:07.690281 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:03:07.690289 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:03:07.690349 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:03:07.691520 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:03:07.691532 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:03:07.693190 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:03:07.693197 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:03:07.695815 (+ 2618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:03:07.695829 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:03:07.697877 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:03:07.697884 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:03:07.698106 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:07.698146 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 21:03:07.699373 (+ 1227us) client_negotiation.cc:240] Negotiation successful
0929 21:03:07.699478 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7751,"mutex_wait_us":97,"spinlock_wait_cycles":6144,"thread_start_us":7425,"threads_started":1}
I20260929 21:03:07.702709 16009 tablet_service.cc:1467] Processing CreateTablet for tablet 7102658cba8d4c74a779e178860043c9 (DEFAULT_TABLE table=test-table [id=997586d28c60489f97a9847a22753b52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:03:07.702884 16143 tablet_service.cc:1467] Processing CreateTablet for tablet 7102658cba8d4c74a779e178860043c9 (DEFAULT_TABLE table=test-table [id=997586d28c60489f97a9847a22753b52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:03:07.703960 16009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7102658cba8d4c74a779e178860043c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:03:07.703611 16276 tablet_service.cc:1467] Processing CreateTablet for tablet 7102658cba8d4c74a779e178860043c9 (DEFAULT_TABLE table=test-table [id=997586d28c60489f97a9847a22753b52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:03:07.704211 16143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7102658cba8d4c74a779e178860043c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:03:07.704910 16276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7102658cba8d4c74a779e178860043c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:03:07.716003 16359 tablet_bootstrap.cc:492] T 7102658cba8d4c74a779e178860043c9 P 746b2763ccbf45d5874eb98263f37a3f: Bootstrap starting.
I20260929 21:03:07.718593 16360 tablet_bootstrap.cc:492] T 7102658cba8d4c74a779e178860043c9 P 3f66c38d69d34c9194b4d1264eddeb3d: Bootstrap starting.
I20260929 21:03:07.719520 16359 tablet_bootstrap.cc:654] T 7102658cba8d4c74a779e178860043c9 P 746b2763ccbf45d5874eb98263f37a3f: Neither blocks nor log segments found. Creating new log.
I20260929 21:03:07.720773 16361 tablet_bootstrap.cc:492] T 7102658cba8d4c74a779e178860043c9 P e5726fa404854fc4ba03654f587816d8: Bootstrap starting.
I20260929 21:03:07.721169 16359 log.cc:824] T 7102658cba8d4c74a779e178860043c9 P 746b2763ccbf45d5874eb98263f37a3f: Log is configured to *not* fsync() on all Append() calls
I20260929 21:03:07.722036 16360 tablet_bootstrap.cc:654] T 7102658cba8d4c74a779e178860043c9 P 3f66c38d69d34c9194b4d1264eddeb3d: Neither blocks nor log segments found. Creating new log.
I20260929 21:03:07.723188 16360 log.cc:824] T 7102658cba8d4c74a779e178860043c9 P 3f66c38d69d34c9194b4d1264eddeb3d: Log is configured to *not* fsync() on all Append() calls
I20260929 21:03:07.723860 16361 tablet_bootstrap.cc:654] T 7102658cba8d4c74a779e178860043c9 P e5726fa404854fc4ba03654f587816d8: Neither blocks nor log segments found. Creating new log.
I20260929 21:03:07.724298 16359 tablet_bootstrap.cc:492] T 7102658cba8d4c74a779e178860043c9 P 746b2763ccbf45d5874eb98263f37a3f: No bootstrap required, opened a new log
I20260929 21:03:07.724592 16359 ts_tablet_manager.cc:1397] T 7102658cba8d4c74a779e178860043c9 P 746b2763ccbf45d5874eb98263f37a3f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 21:03:07.725033 16361 log.cc:824] T 7102658cba8d4c74a779e178860043c9 P e5726fa404854fc4ba03654f587816d8: Log is configured to *not* fsync() on all Append() calls
I20260929 21:03:07.726055 16360 tablet_bootstrap.cc:492] T 7102658cba8d4c74a779e178860043c9 P 3f66c38d69d34c9194b4d1264eddeb3d: No bootstrap required, opened a new log
I20260929 21:03:07.726375 16360 ts_tablet_manager.cc:1397] T 7102658cba8d4c74a779e178860043c9 P 3f66c38d69d34c9194b4d1264eddeb3d: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 21:03:07.727665 16361 tablet_bootstrap.cc:492] T 7102658cba8d4c74a779e178860043c9 P e5726fa404854fc4ba03654f587816d8: No bootstrap required, opened a new log
I20260929 21:03:07.727908 16361 ts_tablet_manager.cc:1397] T 7102658cba8d4c74a779e178860043c9 P e5726fa404854fc4ba03654f587816d8: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260929 21:03:07.735437 16359 raft_consensus.cc:348] T 7102658cba8d4c74a779e178860043c9 P 746b2763ccbf45d5874eb98263f37a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746b2763ccbf45d5874eb98263f37a3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46765 } } peers { permanent_uuid: "3f66c38d69d34c9194b4d1264eddeb3d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36811 } } peers { permanent_uuid: "e5726fa404854fc4ba03654f587816d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33721 } }
I20260929 21:03:07.735978 16359 raft_consensus.cc:374] T 7102658cba8d4c74a779e178860043c9 P 746b2763ccbf45d5874eb98263f37a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:03:07.735760 16361 raft_consensus.cc:348] T 7102658cba8d4c74a779e178860043c9 P e5726fa404854fc4ba03654f587816d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746b2763ccbf45d5874eb98263f37a3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46765 } } peers { permanent_uuid: "3f66c38d69d34c9194b4d1264eddeb3d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36811 } } peers { permanent_uuid: "e5726fa404854fc4ba03654f587816d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33721 } }
I20260929 21:03:07.736084 16359 raft_consensus.cc:729] T 7102658cba8d4c74a779e178860043c9 P 746b2763ccbf45d5874eb98263f37a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 746b2763ccbf45d5874eb98263f37a3f, State: Initialized, Role: FOLLOWER
I20260929 21:03:07.736210 16361 raft_consensus.cc:374] T 7102658cba8d4c74a779e178860043c9 P e5726fa404854fc4ba03654f587816d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:03:07.736323 16361 raft_consensus.cc:729] T 7102658cba8d4c74a779e178860043c9 P e5726fa404854fc4ba03654f587816d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5726fa404854fc4ba03654f587816d8, State: Initialized, Role: FOLLOWER
I20260929 21:03:07.736819 16359 consensus_queue.cc:260] T 7102658cba8d4c74a779e178860043c9 P 746b2763ccbf45d5874eb98263f37a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746b2763ccbf45d5874eb98263f37a3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46765 } } peers { permanent_uuid: "3f66c38d69d34c9194b4d1264eddeb3d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36811 } } peers { permanent_uuid: "e5726fa404854fc4ba03654f587816d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33721 } }
I20260929 21:03:07.736987 16361 consensus_queue.cc:260] T 7102658cba8d4c74a779e178860043c9 P e5726fa404854fc4ba03654f587816d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746b2763ccbf45d5874eb98263f37a3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46765 } } peers { permanent_uuid: "3f66c38d69d34c9194b4d1264eddeb3d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36811 } } peers { permanent_uuid: "e5726fa404854fc4ba03654f587816d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33721 } }
I20260929 21:03:07.736984 16360 raft_consensus.cc:348] T 7102658cba8d4c74a779e178860043c9 P 3f66c38d69d34c9194b4d1264eddeb3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746b2763ccbf45d5874eb98263f37a3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46765 } } peers { permanent_uuid: "3f66c38d69d34c9194b4d1264eddeb3d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36811 } } peers { permanent_uuid: "e5726fa404854fc4ba03654f587816d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33721 } }
I20260929 21:03:07.737414 16360 raft_consensus.cc:374] T 7102658cba8d4c74a779e178860043c9 P 3f66c38d69d34c9194b4d1264eddeb3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:03:07.737602 16360 raft_consensus.cc:729] T 7102658cba8d4c74a779e178860043c9 P 3f66c38d69d34c9194b4d1264eddeb3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f66c38d69d34c9194b4d1264eddeb3d, State: Initialized, Role: FOLLOWER
I20260929 21:03:07.738209 16208 heartbeater.cc:503] Master 127.0.97.62:46807 was elected leader, sending a full tablet report...
I20260929 21:03:07.738466 16360 consensus_queue.cc:260] T 7102658cba8d4c74a779e178860043c9 P 3f66c38d69d34c9194b4d1264eddeb3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746b2763ccbf45d5874eb98263f37a3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46765 } } peers { permanent_uuid: "3f66c38d69d34c9194b4d1264eddeb3d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36811 } } peers { permanent_uuid: "e5726fa404854fc4ba03654f587816d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33721 } }
I20260929 21:03:07.738967 16361 ts_tablet_manager.cc:1428] T 7102658cba8d4c74a779e178860043c9 P e5726fa404854fc4ba03654f587816d8: Time spent starting tablet: real 0.011s user 0.012s sys 0.001s
I20260929 21:03:07.745281 16359 ts_tablet_manager.cc:1428] T 7102658cba8d4c74a779e178860043c9 P 746b2763ccbf45d5874eb98263f37a3f: Time spent starting tablet: real 0.021s user 0.008s sys 0.011s
I20260929 21:03:07.747764 16341 heartbeater.cc:503] Master 127.0.97.62:46807 was elected leader, sending a full tablet report...
I20260929 21:03:07.748337 16360 ts_tablet_manager.cc:1428] T 7102658cba8d4c74a779e178860043c9 P 3f66c38d69d34c9194b4d1264eddeb3d: Time spent starting tablet: real 0.022s user 0.013s sys 0.008s
W20260929 21:03:07.846598 16075 tablet.cc:2369] T 7102658cba8d4c74a779e178860043c9 P 746b2763ccbf45d5874eb98263f37a3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 21:03:07.858569 16366 raft_consensus.cc:482] T 7102658cba8d4c74a779e178860043c9 P e5726fa404854fc4ba03654f587816d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 21:03:07.858912 16366 raft_consensus.cc:504] T 7102658cba8d4c74a779e178860043c9 P e5726fa404854fc4ba03654f587816d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746b2763ccbf45d5874eb98263f37a3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46765 } } peers { permanent_uuid: "3f66c38d69d34c9194b4d1264eddeb3d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36811 } } peers { permanent_uuid: "e5726fa404854fc4ba03654f587816d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33721 } }
W20260929 21:03:07.859279 16209 tablet.cc:2369] T 7102658cba8d4c74a779e178860043c9 P e5726fa404854fc4ba03654f587816d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 21:03:07.860090 16342 tablet.cc:2369] T 7102658cba8d4c74a779e178860043c9 P 3f66c38d69d34c9194b4d1264eddeb3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 21:03:07.860939 16366 leader_election.cc:288] T 7102658cba8d4c74a779e178860043c9 P e5726fa404854fc4ba03654f587816d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 746b2763ccbf45d5874eb98263f37a3f (127.0.97.1:46765), 3f66c38d69d34c9194b4d1264eddeb3d (127.0.97.3:36811)
I20260929 21:03:07.867596 16356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:07.861387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54643 (local address 127.0.97.3:36811)
0929 21:03:07.861626 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 21:03:07.861636 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 21:03:07.862242 (+ 606us) server_negotiation.cc:407] Connection header received
0929 21:03:07.862478 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:03:07.862490 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:03:07.862549 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:03:07.862643 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:03:07.863217 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:07.863922 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:07.865818 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:07.866613 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:07.866760 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:07.866883 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 21:03:07.867211 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 21:03:07.867335 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 21:03:07.867604 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:07.861304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43007 (local address 127.0.97.1:46765)
0929 21:03:07.861541 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 21:03:07.861549 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:03:07.862355 (+ 806us) server_negotiation.cc:407] Connection header received
0929 21:03:07.862516 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:03:07.862520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:03:07.862577 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:03:07.862673 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:03:07.863271 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:07.863949 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:07.865833 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:07.866636 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:07.866797 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:07.866913 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 21:03:07.867225 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 21:03:07.867351 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 21:03:07.867780 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:07.861211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36811 (local address 127.0.97.2:54643)
0929 21:03:07.862122 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:07.862141 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 21:03:07.862333 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:03:07.862657 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:03:07.862661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:03:07.862685 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:03:07.863072 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:03:07.863080 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:03:07.864042 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:03:07.864051 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:03:07.865665 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:03:07.865677 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:03:07.866839 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:03:07.866848 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:03:07.867053 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:07.867077 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 21:03:07.867183 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 21:03:07.867320 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"mutex_wait_us":48,"thread_start_us":549,"threads_started":1}
I20260929 21:03:07.867841 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:07.861320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46765 (local address 127.0.97.2:43007)
0929 21:03:07.862248 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:07.862263 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 21:03:07.862367 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:03:07.862693 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:03:07.862696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:03:07.862719 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:03:07.863126 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:03:07.863134 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:03:07.864079 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:03:07.864085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:03:07.865687 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:03:07.865699 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:03:07.866865 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:03:07.866871 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:03:07.867068 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:07.867090 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 21:03:07.867195 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 21:03:07.867324 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"mutex_wait_us":75,"thread_start_us":537,"threads_started":1}
I20260929 21:03:07.868986 16296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7102658cba8d4c74a779e178860043c9" candidate_uuid: "e5726fa404854fc4ba03654f587816d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f66c38d69d34c9194b4d1264eddeb3d" is_pre_election: true
I20260929 21:03:07.868953 16029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7102658cba8d4c74a779e178860043c9" candidate_uuid: "e5726fa404854fc4ba03654f587816d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "746b2763ccbf45d5874eb98263f37a3f" is_pre_election: true
I20260929 21:03:07.869449 16029 raft_consensus.cc:2454] T 7102658cba8d4c74a779e178860043c9 P 746b2763ccbf45d5874eb98263f37a3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5726fa404854fc4ba03654f587816d8 in term 0.
I20260929 21:03:07.869449 16296 raft_consensus.cc:2454] T 7102658cba8d4c74a779e178860043c9 P 3f66c38d69d34c9194b4d1264eddeb3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5726fa404854fc4ba03654f587816d8 in term 0.
I20260929 21:03:07.870059 16098 leader_election.cc:302] T 7102658cba8d4c74a779e178860043c9 P e5726fa404854fc4ba03654f587816d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f66c38d69d34c9194b4d1264eddeb3d, e5726fa404854fc4ba03654f587816d8; no voters:
I20260929 21:03:07.870457 16366 raft_consensus.cc:2781] T 7102658cba8d4c74a779e178860043c9 P e5726fa404854fc4ba03654f587816d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 21:03:07.870599 16366 raft_consensus.cc:482] T 7102658cba8d4c74a779e178860043c9 P e5726fa404854fc4ba03654f587816d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 21:03:07.870723 16366 raft_consensus.cc:3037] T 7102658cba8d4c74a779e178860043c9 P e5726fa404854fc4ba03654f587816d8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:03:07.872813 16366 raft_consensus.cc:504] T 7102658cba8d4c74a779e178860043c9 P e5726fa404854fc4ba03654f587816d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746b2763ccbf45d5874eb98263f37a3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46765 } } peers { permanent_uuid: "3f66c38d69d34c9194b4d1264eddeb3d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36811 } } peers { permanent_uuid: "e5726fa404854fc4ba03654f587816d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33721 } }
I20260929 21:03:07.873430 16366 leader_election.cc:288] T 7102658cba8d4c74a779e178860043c9 P e5726fa404854fc4ba03654f587816d8 [CANDIDATE]: Term 1 election: Requested vote from peers 746b2763ccbf45d5874eb98263f37a3f (127.0.97.1:46765), 3f66c38d69d34c9194b4d1264eddeb3d (127.0.97.3:36811)
I20260929 21:03:07.873870 16029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7102658cba8d4c74a779e178860043c9" candidate_uuid: "e5726fa404854fc4ba03654f587816d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "746b2763ccbf45d5874eb98263f37a3f"
I20260929 21:03:07.873934 16296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7102658cba8d4c74a779e178860043c9" candidate_uuid: "e5726fa404854fc4ba03654f587816d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f66c38d69d34c9194b4d1264eddeb3d"
I20260929 21:03:07.874053 16029 raft_consensus.cc:3037] T 7102658cba8d4c74a779e178860043c9 P 746b2763ccbf45d5874eb98263f37a3f [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:03:07.874178 16296 raft_consensus.cc:3037] T 7102658cba8d4c74a779e178860043c9 P 3f66c38d69d34c9194b4d1264eddeb3d [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:03:07.876227 16029 raft_consensus.cc:2454] T 7102658cba8d4c74a779e178860043c9 P 746b2763ccbf45d5874eb98263f37a3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5726fa404854fc4ba03654f587816d8 in term 1.
I20260929 21:03:07.876366 16296 raft_consensus.cc:2454] T 7102658cba8d4c74a779e178860043c9 P 3f66c38d69d34c9194b4d1264eddeb3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5726fa404854fc4ba03654f587816d8 in term 1.
I20260929 21:03:07.876700 16097 leader_election.cc:302] T 7102658cba8d4c74a779e178860043c9 P e5726fa404854fc4ba03654f587816d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 746b2763ccbf45d5874eb98263f37a3f, e5726fa404854fc4ba03654f587816d8; no voters:
I20260929 21:03:07.877074 16366 raft_consensus.cc:2781] T 7102658cba8d4c74a779e178860043c9 P e5726fa404854fc4ba03654f587816d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:03:07.877564 16366 raft_consensus.cc:686] T 7102658cba8d4c74a779e178860043c9 P e5726fa404854fc4ba03654f587816d8 [term 1 LEADER]: Becoming Leader. State: Replica: e5726fa404854fc4ba03654f587816d8, State: Running, Role: LEADER
I20260929 21:03:07.878098 16366 consensus_queue.cc:237] T 7102658cba8d4c74a779e178860043c9 P e5726fa404854fc4ba03654f587816d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746b2763ccbf45d5874eb98263f37a3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46765 } } peers { permanent_uuid: "3f66c38d69d34c9194b4d1264eddeb3d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36811 } } peers { permanent_uuid: "e5726fa404854fc4ba03654f587816d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33721 } }
I20260929 21:03:07.882491 15884 catalog_manager.cc:5495] T 7102658cba8d4c74a779e178860043c9 P e5726fa404854fc4ba03654f587816d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to e5726fa404854fc4ba03654f587816d8 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "e5726fa404854fc4ba03654f587816d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746b2763ccbf45d5874eb98263f37a3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f66c38d69d34c9194b4d1264eddeb3d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5726fa404854fc4ba03654f587816d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33721 } health_report { overall_health: HEALTHY } } }
I20260929 21:03:07.897123 16358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:07.892243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49632 (local address 127.0.97.2:33721)
0929 21:03:07.892412 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 21:03:07.892416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:03:07.892432 (+ 16us) server_negotiation.cc:407] Connection header received
0929 21:03:07.892501 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:03:07.892504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:03:07.892573 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:03:07.892687 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:03:07.893060 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:07.893743 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:07.895114 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:07.895514 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:07.895613 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:07.896033 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:03:07.896504 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:03:07.896613 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 21:03:07.896708 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 21:03:07.896768 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 21:03:07.905184 16029 raft_consensus.cc:1261] T 7102658cba8d4c74a779e178860043c9 P 746b2763ccbf45d5874eb98263f37a3f [term 1 FOLLOWER]: Refusing update from remote peer e5726fa404854fc4ba03654f587816d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:03:07.905180 16296 raft_consensus.cc:1261] T 7102658cba8d4c74a779e178860043c9 P 3f66c38d69d34c9194b4d1264eddeb3d [term 1 FOLLOWER]: Refusing update from remote peer e5726fa404854fc4ba03654f587816d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:03:07.906368 16366 consensus_queue.cc:1035] T 7102658cba8d4c74a779e178860043c9 P e5726fa404854fc4ba03654f587816d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "746b2763ccbf45d5874eb98263f37a3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:03:07.906667 16370 consensus_queue.cc:1035] T 7102658cba8d4c74a779e178860043c9 P e5726fa404854fc4ba03654f587816d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f66c38d69d34c9194b4d1264eddeb3d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:03:07.918344 16373 mvcc.cc:205] Tried to move back new op lower bound from 7334771867248799744 to 7334771867152457728. Current Snapshot: MvccSnapshot[applied={T|T < 7334771867248799744}]
I20260929 21:03:07.930037 15885 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 21:03:07.930348 15885 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 21:03:07.934016 15885 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21: Sending DeleteTablet for 3 replicas of tablet 7102658cba8d4c74a779e178860043c9
I20260929 21:03:07.935072 16009 tablet_service.cc:1514] Processing DeleteTablet for tablet 7102658cba8d4c74a779e178860043c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:03:07 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60726
I20260929 21:03:07.935325 16276 tablet_service.cc:1514] Processing DeleteTablet for tablet 7102658cba8d4c74a779e178860043c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:03:07 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41816
I20260929 21:03:07.935475 16143 tablet_service.cc:1514] Processing DeleteTablet for tablet 7102658cba8d4c74a779e178860043c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:03:07 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49620
I20260929 21:03:07.936614 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 15946
I20260929 21:03:07.949370 16385 tablet_replica.cc:330] T 7102658cba8d4c74a779e178860043c9 P e5726fa404854fc4ba03654f587816d8: stopping tablet replica
I20260929 21:03:07.949544 16383 tablet_replica.cc:330] T 7102658cba8d4c74a779e178860043c9 P 746b2763ccbf45d5874eb98263f37a3f: stopping tablet replica
I20260929 21:03:07.949887 16383 raft_consensus.cc:2229] T 7102658cba8d4c74a779e178860043c9 P 746b2763ccbf45d5874eb98263f37a3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:03:07.949900 16385 raft_consensus.cc:2229] T 7102658cba8d4c74a779e178860043c9 P e5726fa404854fc4ba03654f587816d8 [term 1 LEADER]: Raft consensus shutting down.
I20260929 21:03:07.950461 16383 raft_consensus.cc:2258] T 7102658cba8d4c74a779e178860043c9 P 746b2763ccbf45d5874eb98263f37a3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:03:07.950590 16385 raft_consensus.cc:2258] T 7102658cba8d4c74a779e178860043c9 P e5726fa404854fc4ba03654f587816d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:03:07.953336 16383 ts_tablet_manager.cc:1905] T 7102658cba8d4c74a779e178860043c9 P 746b2763ccbf45d5874eb98263f37a3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:03:07.954176 16385 ts_tablet_manager.cc:1905] T 7102658cba8d4c74a779e178860043c9 P e5726fa404854fc4ba03654f587816d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:03:07.961400 16383 ts_tablet_manager.cc:1918] T 7102658cba8d4c74a779e178860043c9 P 746b2763ccbf45d5874eb98263f37a3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:03:07.961601 16385 ts_tablet_manager.cc:1918] T 7102658cba8d4c74a779e178860043c9 P e5726fa404854fc4ba03654f587816d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:03:07.961692 16383 log.cc:1196] T 7102658cba8d4c74a779e178860043c9 P 746b2763ccbf45d5874eb98263f37a3f: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/wal/wals/7102658cba8d4c74a779e178860043c9
I20260929 21:03:07.961835 16385 log.cc:1196] T 7102658cba8d4c74a779e178860043c9 P e5726fa404854fc4ba03654f587816d8: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/wal/wals/7102658cba8d4c74a779e178860043c9
I20260929 21:03:07.962448 16383 ts_tablet_manager.cc:1939] T 7102658cba8d4c74a779e178860043c9 P 746b2763ccbf45d5874eb98263f37a3f: Deleting consensus metadata
I20260929 21:03:07.962462 16385 ts_tablet_manager.cc:1939] T 7102658cba8d4c74a779e178860043c9 P e5726fa404854fc4ba03654f587816d8: Deleting consensus metadata
Sep 29 21:03:07 dist-test-slave-1dms krb5kdc[15840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 21:03:07.964506 15871 catalog_manager.cc:4841] TS e5726fa404854fc4ba03654f587816d8 (127.0.97.2:33721): tablet 7102658cba8d4c74a779e178860043c9 (table test-table [id=997586d28c60489f97a9847a22753b52]) successfully deleted
I20260929 21:03:07.965165 15870 catalog_manager.cc:4841] TS 746b2763ccbf45d5874eb98263f37a3f (127.0.97.1:46765): tablet 7102658cba8d4c74a779e178860043c9 (table test-table [id=997586d28c60489f97a9847a22753b52]) successfully deleted
I20260929 21:03:07.965442 16384 tablet_replica.cc:330] T 7102658cba8d4c74a779e178860043c9 P 3f66c38d69d34c9194b4d1264eddeb3d: stopping tablet replica
I20260929 21:03:07.965845 16384 raft_consensus.cc:2229] T 7102658cba8d4c74a779e178860043c9 P 3f66c38d69d34c9194b4d1264eddeb3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:03:07.966305 16384 raft_consensus.cc:2258] T 7102658cba8d4c74a779e178860043c9 P 3f66c38d69d34c9194b4d1264eddeb3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:03:07.968415 16384 ts_tablet_manager.cc:1905] T 7102658cba8d4c74a779e178860043c9 P 3f66c38d69d34c9194b4d1264eddeb3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:03:07.973876 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:07.938106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60742 (local address 127.0.97.1:46765)
0929 21:03:07.938338 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 21:03:07.938348 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 21:03:07.944567 (+ 6219us) server_negotiation.cc:407] Connection header received
0929 21:03:07.945352 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:03:07.945362 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:03:07.945505 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:03:07.945603 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:03:07.952753 (+ 7150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:07.953867 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:07.957852 (+ 3985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:07.958545 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:07.958665 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:07.967931 (+ 9266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:03:07.967943 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:03:07.967956 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:03:07.967997 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:03:07.970950 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:03:07.971672 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:03:07.971679 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:03:07.971681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:03:07.971813 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:03:07.972214 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:03:07.972218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:03:07.972220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:03:07.972554 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:03:07.972893 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0929 21:03:07.973265 (+ 372us) server_negotiation.cc:299] Negotiation successful
0929 21:03:07.973496 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 21:03:07.974575 16069 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:03:07.975476 16384 ts_tablet_manager.cc:1918] T 7102658cba8d4c74a779e178860043c9 P 3f66c38d69d34c9194b4d1264eddeb3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:03:07.975677 16384 log.cc:1196] T 7102658cba8d4c74a779e178860043c9 P 3f66c38d69d34c9194b4d1264eddeb3d: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/wal/wals/7102658cba8d4c74a779e178860043c9
I20260929 21:03:07.976246 16384 ts_tablet_manager.cc:1939] T 7102658cba8d4c74a779e178860043c9 P 3f66c38d69d34c9194b4d1264eddeb3d: Deleting consensus metadata
I20260929 21:03:07.978531 15871 catalog_manager.cc:4841] TS 3f66c38d69d34c9194b4d1264eddeb3d (127.0.97.3:36811): tablet 7102658cba8d4c74a779e178860043c9 (table test-table [id=997586d28c60489f97a9847a22753b52]) successfully deleted
I20260929 21:03:08.078996 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 15946
I20260929 21:03:08.097955 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 16080
Sep 29 21:03:08 dist-test-slave-1dms krb5kdc[15840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 21:03:08.114724 16358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:08.098760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49636 (local address 127.0.97.2:33721)
0929 21:03:08.098969 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 21:03:08.098980 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 21:03:08.099003 (+ 23us) server_negotiation.cc:407] Connection header received
0929 21:03:08.099103 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:03:08.099109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:03:08.099190 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:03:08.099296 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:03:08.100538 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:08.101759 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:08.103101 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:08.103736 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:08.103887 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:08.108330 (+ 4443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:03:08.108341 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:03:08.108354 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:03:08.108390 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:03:08.111784 (+ 3394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:03:08.112451 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:03:08.112466 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:03:08.112468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:03:08.112594 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:03:08.113171 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:03:08.113176 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:03:08.113190 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:03:08.113520 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:03:08.113896 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0929 21:03:08.114132 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 21:03:08.114347 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 21:03:08.115427 16203 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:03:08.230406 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 16080
I20260929 21:03:08.249276 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 16213
Sep 29 21:03:08 dist-test-slave-1dms krb5kdc[15840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 21:03:08.263841 16356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:08.250033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41820 (local address 127.0.97.3:36811)
0929 21:03:08.250212 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 21:03:08.250220 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:03:08.250258 (+ 38us) server_negotiation.cc:407] Connection header received
0929 21:03:08.250464 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:03:08.250474 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:03:08.250550 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:03:08.250659 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:03:08.251904 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:08.252626 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:08.254070 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:08.254456 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:08.254589 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:08.258945 (+ 4356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:03:08.258959 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:03:08.258975 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:03:08.259007 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:03:08.261563 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:03:08.262169 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:03:08.262176 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:03:08.262178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:03:08.262307 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:03:08.262611 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:03:08.262614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:03:08.262616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:03:08.262939 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:03:08.263149 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 21:03:08.263437 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 21:03:08.263596 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 21:03:08.264518 16336 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:03:08.354005 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 16213
I20260929 21:03:08.370658 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 15853
I20260929 21:03:08.371802 15915 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:03:08.483560 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 15853
I20260929 21:03:28.509913 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46807
--webserver_interface=127.0.97.62
--webserver_port=43369
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:03:28.838586 16391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:03:28.838923 16391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:03:28.839092 16391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:03:28.846395 16391 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 21:03:28.846524 16391 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 21:03:28.846618 16391 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 21:03:28.846717 16391 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:03:28.846825 16391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:03:28.846946 16391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:03:28.847056 16391 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:03:28.847136 16391 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:03:28.856887 16391 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=43369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:03:28.858314 16391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:03:28.859620 16391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:03:28.881757 16398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:03:28.881788 16397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:03:28.881826 16400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:03:28.882787 16391 server_base.cc:1034] running on GCE node
I20260929 21:03:28.883708 16391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:03:28.885383 16391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:03:28.886646 16391 hybrid_clock.cc:648] HybridClock initialized: now 1790715808886577 us; error 42 us; skew 500 ppm
I20260929 21:03:28.887053 16391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:03:28 dist-test-slave-1dms krb5kdc[15840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:03:28.890832 16391 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 21:03:28.892598 16391 webserver.cc:460] Webserver started at http://127.0.97.62:43369/ using document root <none> and password file <none>
I20260929 21:03:28.893414 16391 fs_manager.cc:359] Metadata directory not provided
I20260929 21:03:28.893518 16391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:03:28.899359 16391 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 21:03:28.902401 16407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:03:28.903793 16391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:03:28.903957 16391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "885b912376d24bc48fda185253979e21"
format_stamp: "Formatted at 2026-09-29 21:03:05 on dist-test-slave-1dms"
server_key: "516751997f5064282644b29234ab7c4e"
server_key_iv: "649df24baeee8faf33fbf6fc42ba08b4"
server_key_version: "encryptionkey@0"
I20260929 21:03:28.905099 16391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:03:28.925014 16391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:03:28.929563 16391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:03:28.929811 16391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:03:28.948381 16391 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46807
I20260929 21:03:28.948408 16458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46807 every 8 connection(s)
I20260929 21:03:28.949918 16391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:03:28.952309 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 16391
I20260929 21:03:28.953169 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:46765
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46807
--webserver_port=33381
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 21:03:28.957173 16459 sys_catalog.cc:263] Verifying existing consensus state
I20260929 21:03:28.961043 16459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21: Bootstrap starting.
I20260929 21:03:28.980648 16459 log.cc:824] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21: Log is configured to *not* fsync() on all Append() calls
I20260929 21:03:28.992426 16459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 21:03:28.993088 16459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21: Bootstrap complete.
I20260929 21:03:29.002074 16459 raft_consensus.cc:348] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885b912376d24bc48fda185253979e21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46807 } }
I20260929 21:03:29.002916 16459 raft_consensus.cc:729] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 885b912376d24bc48fda185253979e21, State: Initialized, Role: FOLLOWER
I20260929 21:03:29.003522 16459 consensus_queue.cc:260] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885b912376d24bc48fda185253979e21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46807 } }
I20260929 21:03:29.003762 16459 raft_consensus.cc:388] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:03:29.003885 16459 raft_consensus.cc:482] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:03:29.004082 16459 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [term 1 FOLLOWER]: Advancing to term 2
I20260929 21:03:29.007975 16459 raft_consensus.cc:504] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885b912376d24bc48fda185253979e21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46807 } }
I20260929 21:03:29.008525 16459 leader_election.cc:302] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 885b912376d24bc48fda185253979e21; no voters:
I20260929 21:03:29.009262 16459 leader_election.cc:288] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 21:03:29.009431 16463 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 21:03:29.010881 16463 raft_consensus.cc:686] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [term 2 LEADER]: Becoming Leader. State: Replica: 885b912376d24bc48fda185253979e21, State: Running, Role: LEADER
I20260929 21:03:29.011507 16463 consensus_queue.cc:237] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885b912376d24bc48fda185253979e21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46807 } }
I20260929 21:03:29.011766 16459 sys_catalog.cc:564] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:03:29.014001 16465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 885b912376d24bc48fda185253979e21. Latest consensus state: current_term: 2 leader_uuid: "885b912376d24bc48fda185253979e21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885b912376d24bc48fda185253979e21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46807 } } }
I20260929 21:03:29.014460 16465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [sys.catalog]: This master's current role is: LEADER
I20260929 21:03:29.015048 16464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "885b912376d24bc48fda185253979e21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885b912376d24bc48fda185253979e21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46807 } } }
I20260929 21:03:29.015388 16464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [sys.catalog]: This master's current role is: LEADER
I20260929 21:03:29.021826 16469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:03:29.030041 16469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:03:29.031528 16469 catalog_manager.cc:1259] Loaded cluster ID: 0d536ebc009246f2ae6910666b207a89
I20260929 21:03:29.031669 16469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:03:29.035717 16469 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:03:29.037333 16469 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21: Loaded TSK: 0
I20260929 21:03:29.075977 16469 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21: Generated new TSK 1
I20260929 21:03:29.076385 16469 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 21:03:29.304054 16461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:03:29.304384 16461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:03:29.304592 16461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:03:29.312374 16461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:03:29.312594 16461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:03:29.312788 16461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 21:03:29.322916 16461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:46765
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=33381
--tserver_master_addrs=127.0.97.62:46807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:03:29.324259 16461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:03:29.325702 16461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:03:29.340415 16488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:03:29.340948 16490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:03:29.341008 16487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:03:29.341702 16461 server_base.cc:1034] running on GCE node
I20260929 21:03:29.342384 16461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:03:29.343573 16461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:03:29.344817 16461 hybrid_clock.cc:648] HybridClock initialized: now 1790715809344748 us; error 77 us; skew 500 ppm
I20260929 21:03:29.345217 16461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:03:29 dist-test-slave-1dms krb5kdc[15840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:03:29.355245 16461 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 21:03:29.356917 16461 webserver.cc:460] Webserver started at http://127.0.97.1:33381/ using document root <none> and password file <none>
I20260929 21:03:29.357633 16461 fs_manager.cc:359] Metadata directory not provided
I20260929 21:03:29.357771 16461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:03:29.363395 16461 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 21:03:29.366199 16497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:03:29.367624 16461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:03:29.367786 16461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/wal
uuid: "746b2763ccbf45d5874eb98263f37a3f"
format_stamp: "Formatted at 2026-09-29 21:03:06 on dist-test-slave-1dms"
server_key: "34cd18bd130d4157d17121a84ce76ee5"
server_key_iv: "cf2f5f75793e972df16db7d45331bc91"
server_key_version: "encryptionkey@0"
I20260929 21:03:29.368731 16461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:03:29.389976 16461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:03:29.394250 16461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:03:29.394529 16461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:03:29.395610 16461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:03:29.397096 16461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:03:29.397190 16461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:03:29.397295 16461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:03:29.397398 16461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:03:29.430681 16461 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46765
I20260929 21:03:29.430773 16609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46765 every 8 connection(s)
I20260929 21:03:29.432163 16461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
I20260929 21:03:29.437288 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 16461
I20260929 21:03:29.437894 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:33721
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46807
--webserver_port=37831
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 21:03:29 dist-test-slave-1dms krb5kdc[15840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:03:29.459352 16612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:29.435062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60925 (local address 127.0.97.62:46807)
0929 21:03:29.435645 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0929 21:03:29.435657 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 21:03:29.436260 (+ 603us) server_negotiation.cc:407] Connection header received
0929 21:03:29.438685 (+ 2425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:03:29.438707 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:03:29.439213 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:03:29.439614 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:03:29.441091 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:29.443192 (+ 2101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:29.444781 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:29.445874 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:29.446022 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:29.450836 (+ 4814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:03:29.450844 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:03:29.450861 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:03:29.450902 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:03:29.454087 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:03:29.455203 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:03:29.455210 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:03:29.455216 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:03:29.455392 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:03:29.455966 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:03:29.455970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:03:29.455972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:03:29.456500 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:03:29.456650 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 21:03:29.457807 (+ 1157us) server_negotiation.cc:299] Negotiation successful
0929 21:03:29.458147 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":258,"threads_started":1}
I20260929 21:03:29.459738 16613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:29.435214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.1:60925)
0929 21:03:29.436070 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:29.436122 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 21:03:29.438009 (+ 1887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:03:29.439746 (+ 1737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:03:29.439753 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:03:29.440301 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:03:29.440815 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:03:29.440884 (+ 69us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:03:29.443384 (+ 2500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:03:29.443388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:03:29.444625 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:03:29.444638 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:03:29.446138 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:03:29.446147 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:03:29.446467 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:29.447244 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:03:29.447284 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:03:29.450500 (+ 3216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:03:29.454260 (+ 3760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:03:29.454273 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:03:29.454295 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:03:29.455002 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:03:29.455543 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:03:29.455551 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:03:29.455558 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:03:29.455833 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:03:29.456648 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:03:29.456658 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:03:29.457206 (+ 548us) client_negotiation.cc:769] Sending connection context
0929 21:03:29.457563 (+ 357us) client_negotiation.cc:240] Negotiation successful
0929 21:03:29.458257 (+ 694us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":362,"threads_started":1}
I20260929 21:03:29.463116 16610 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46807
I20260929 21:03:29.463445 16610 heartbeater.cc:464] Registering TS with master...
I20260929 21:03:29.464179 16610 heartbeater.cc:511] Master 127.0.97.62:46807 requested a full tablet report, sending...
I20260929 21:03:29.466394 16424 ts_manager.cc:194] Registered new tserver with Master: 746b2763ccbf45d5874eb98263f37a3f (127.0.97.1:46765)
I20260929 21:03:29.468026 16424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:60925
W20260929 21:03:29.765043 16614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:03:29.765355 16614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:03:29.765516 16614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:03:29.772612 16614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:03:29.772747 16614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:03:29.772945 16614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 21:03:29.782194 16614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:33721
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=37831
--tserver_master_addrs=127.0.97.62:46807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:03:29.783365 16614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:03:29.784583 16614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:03:29.797190 16619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:03:29.797147 16622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:03:29.797243 16620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:03:29.798084 16614 server_base.cc:1034] running on GCE node
I20260929 21:03:29.798642 16614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:03:29.799551 16614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:03:29.800769 16614 hybrid_clock.cc:648] HybridClock initialized: now 1790715809800729 us; error 55 us; skew 500 ppm
I20260929 21:03:29.801095 16614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:03:29 dist-test-slave-1dms krb5kdc[15840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:03:29.804028 16614 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 21:03:29.805433 16614 webserver.cc:460] Webserver started at http://127.0.97.2:37831/ using document root <none> and password file <none>
I20260929 21:03:29.806082 16614 fs_manager.cc:359] Metadata directory not provided
I20260929 21:03:29.806246 16614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:03:29.811340 16614 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 21:03:29.814066 16629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:03:29.815321 16614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:03:29.815488 16614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/wal
uuid: "e5726fa404854fc4ba03654f587816d8"
format_stamp: "Formatted at 2026-09-29 21:03:07 on dist-test-slave-1dms"
server_key: "7cad1c146ac263cca6b5a42bf890ae28"
server_key_iv: "42dc0d4024cce989637d4b869183890f"
server_key_version: "encryptionkey@0"
I20260929 21:03:29.816406 16614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:03:29.840512 16614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:03:29.844290 16614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:03:29.844591 16614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:03:29.845682 16614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:03:29.846997 16614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:03:29.847121 16614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:03:29.847245 16614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:03:29.847357 16614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:03:29.879797 16614 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33721
I20260929 21:03:29.879878 16741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33721 every 8 connection(s)
I20260929 21:03:29.881286 16614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
I20260929 21:03:29.883598 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 16614
I20260929 21:03:29.884300 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:36811
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46807
--webserver_port=36963
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 21:03:29 dist-test-slave-1dms krb5kdc[15840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:03:29.903353 16612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:29.883844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39277 (local address 127.0.97.62:46807)
0929 21:03:29.884082 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 21:03:29.884088 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:03:29.885601 (+ 1513us) server_negotiation.cc:407] Connection header received
0929 21:03:29.887437 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:03:29.887446 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:03:29.887543 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:03:29.887668 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:03:29.889134 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:29.890267 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:29.891771 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:29.892194 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:29.892413 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:29.896067 (+ 3654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:03:29.896076 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:03:29.896083 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:03:29.896130 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:03:29.899717 (+ 3587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:03:29.900452 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:03:29.900460 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:03:29.900466 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:03:29.900616 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:03:29.901189 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:03:29.901197 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:03:29.901203 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:03:29.901616 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:03:29.901783 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 21:03:29.902610 (+ 827us) server_negotiation.cc:299] Negotiation successful
0929 21:03:29.902894 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 21:03:29.903879 16744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:29.884245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.2:39277)
0929 21:03:29.885423 (+ 1178us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:29.885475 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 21:03:29.887155 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:03:29.887870 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:03:29.887878 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:03:29.888365 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:03:29.888972 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:03:29.888989 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:03:29.890416 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:03:29.890420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:03:29.891620 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:03:29.891631 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:03:29.892482 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:03:29.892490 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:03:29.892715 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:29.893439 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:03:29.893464 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:03:29.895830 (+ 2366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:03:29.899860 (+ 4030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:03:29.899866 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:03:29.899878 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:03:29.900307 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:03:29.900786 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:03:29.900790 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:03:29.900793 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:03:29.901056 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:03:29.901747 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:03:29.901752 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:03:29.902072 (+ 320us) client_negotiation.cc:769] Sending connection context
0929 21:03:29.902334 (+ 262us) client_negotiation.cc:240] Negotiation successful
0929 21:03:29.902714 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":908,"thread_start_us":723,"threads_started":1}
I20260929 21:03:29.905545 16742 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46807
I20260929 21:03:29.905877 16742 heartbeater.cc:464] Registering TS with master...
I20260929 21:03:29.906613 16742 heartbeater.cc:511] Master 127.0.97.62:46807 requested a full tablet report, sending...
I20260929 21:03:29.908313 16424 ts_manager.cc:194] Registered new tserver with Master: e5726fa404854fc4ba03654f587816d8 (127.0.97.2:33721)
I20260929 21:03:29.909716 16424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39277
W20260929 21:03:30.221335 16745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:03:30.221592 16745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:03:30.221683 16745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:03:30.228971 16745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:03:30.229072 16745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:03:30.229184 16745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 21:03:30.238611 16745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:36811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=36963
--tserver_master_addrs=127.0.97.62:46807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:03:30.239781 16745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:03:30.241098 16745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:03:30.253674 16750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:03:30.253714 16753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:03:30.253801 16751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:03:30.254824 16745 server_base.cc:1034] running on GCE node
I20260929 21:03:30.255496 16745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:03:30.256479 16745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:03:30.257695 16745 hybrid_clock.cc:648] HybridClock initialized: now 1790715810257650 us; error 38 us; skew 500 ppm
I20260929 21:03:30.258033 16745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:03:30 dist-test-slave-1dms krb5kdc[15840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:03:30.261262 16745 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 21:03:30.263114 16745 webserver.cc:460] Webserver started at http://127.0.97.3:36963/ using document root <none> and password file <none>
I20260929 21:03:30.263787 16745 fs_manager.cc:359] Metadata directory not provided
I20260929 21:03:30.263947 16745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:03:30.269255 16745 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 21:03:30.271977 16760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:03:30.273468 16745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:03:30.273646 16745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/wal
uuid: "3f66c38d69d34c9194b4d1264eddeb3d"
format_stamp: "Formatted at 2026-09-29 21:03:07 on dist-test-slave-1dms"
server_key: "b0facff35d5cf4a30c4a791c4c936e1d"
server_key_iv: "3944b588241ddd381526015d31495ff4"
server_key_version: "encryptionkey@0"
I20260929 21:03:30.274629 16745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:03:30.297729 16745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:03:30.302095 16745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:03:30.302371 16745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:03:30.303336 16745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:03:30.304620 16745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:03:30.304755 16745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:03:30.304924 16745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:03:30.305027 16745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:03:30.338476 16745 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36811
I20260929 21:03:30.338572 16872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36811 every 8 connection(s)
I20260929 21:03:30.339895 16745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
I20260929 21:03:30.349989 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 16745
Sep 29 21:03:30 dist-test-slave-1dms krb5kdc[15840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:03:30.360747 16612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:30.342451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41581 (local address 127.0.97.62:46807)
0929 21:03:30.342691 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 21:03:30.342698 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:03:30.343778 (+ 1080us) server_negotiation.cc:407] Connection header received
0929 21:03:30.345649 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:03:30.345653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:03:30.345717 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:03:30.345795 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:03:30.347270 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:30.347974 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:30.349671 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:30.350328 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:30.350467 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:30.354087 (+ 3620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:03:30.354095 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:03:30.354101 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:03:30.354145 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:03:30.357332 (+ 3187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:03:30.358234 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:03:30.358241 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:03:30.358246 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:03:30.358390 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:03:30.358859 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:03:30.358862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:03:30.358864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:03:30.359177 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:03:30.359287 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 21:03:30.360127 (+ 840us) server_negotiation.cc:299] Negotiation successful
0929 21:03:30.360276 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 21:03:30.361740 16875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:30.342922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.3:41581)
0929 21:03:30.343614 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:30.343661 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 21:03:30.345392 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:03:30.346045 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:03:30.346053 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:03:30.346571 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:03:30.347126 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:03:30.347143 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:03:30.348171 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:03:30.348175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:03:30.349532 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:03:30.349544 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:03:30.350537 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:03:30.350542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:03:30.350879 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:30.351650 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:03:30.351684 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:03:30.353891 (+ 2207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:03:30.357488 (+ 3597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:03:30.357495 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:03:30.357511 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:03:30.358114 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:03:30.358508 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:03:30.358515 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:03:30.358521 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:03:30.358744 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:03:30.359310 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:03:30.359320 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:03:30.359744 (+ 424us) client_negotiation.cc:769] Sending connection context
0929 21:03:30.360059 (+ 315us) client_negotiation.cc:240] Negotiation successful
0929 21:03:30.360579 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":277,"threads_started":1}
I20260929 21:03:30.363145 16873 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46807
I20260929 21:03:30.363441 16873 heartbeater.cc:464] Registering TS with master...
I20260929 21:03:30.364297 16873 heartbeater.cc:511] Master 127.0.97.62:46807 requested a full tablet report, sending...
I20260929 21:03:30.365950 16424 ts_manager.cc:194] Registered new tserver with Master: 3f66c38d69d34c9194b4d1264eddeb3d (127.0.97.3:36811)
I20260929 21:03:30.366554 16876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:30.350959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38084 (local address 127.0.97.62:46807)
0929 21:03:30.351513 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0929 21:03:30.351521 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:03:30.351950 (+ 429us) server_negotiation.cc:407] Connection header received
0929 21:03:30.352129 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:03:30.352140 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:03:30.352194 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:03:30.352301 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:03:30.353925 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:30.355150 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:30.356636 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:30.357352 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:30.357494 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:30.360754 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:03:30.360761 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:03:30.360769 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:03:30.360806 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:03:30.363645 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:03:30.364463 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:03:30.364468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:03:30.364472 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:03:30.364584 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:03:30.365118 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:03:30.365125 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:03:30.365130 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:03:30.365475 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:03:30.365717 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 21:03:30.366019 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 21:03:30.366222 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":275,"threads_started":1}
I20260929 21:03:30.367139 16424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41581
I20260929 21:03:30.368970 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 21:03:30.375545 16876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:30.369822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38090 (local address 127.0.97.62:46807)
0929 21:03:30.369996 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 21:03:30.370003 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:03:30.370395 (+ 392us) server_negotiation.cc:407] Connection header received
0929 21:03:30.370637 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:03:30.370641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:03:30.370762 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:03:30.370913 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:03:30.371318 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:30.372071 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:30.373660 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:30.374090 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:30.374215 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:30.374711 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:03:30.374906 (+ 195us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 21:03:30.375257 (+ 351us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38090: token expired
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 21:03:30.375662 16876 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38090: token expired
W20260929 21:03:30.376139 16346 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:46807)
I20260929 21:03:30.387933 16876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:30.376585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38096 (local address 127.0.97.62:46807)
0929 21:03:30.376724 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 21:03:30.376727 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 21:03:30.376742 (+ 15us) server_negotiation.cc:407] Connection header received
0929 21:03:30.376890 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:03:30.376901 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:03:30.376957 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:03:30.377030 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:03:30.378306 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:30.379080 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:30.380347 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:30.380746 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:30.380901 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:30.383355 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:03:30.383359 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:03:30.383362 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:03:30.383392 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:03:30.385782 (+ 2390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:03:30.386428 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:03:30.386432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:03:30.386434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:03:30.386549 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:03:30.386858 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:03:30.386861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:03:30.386863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:03:30.387141 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:03:30.387317 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 21:03:30.387541 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 21:03:30.387700 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 21:03:30.391199 16424 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38096:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 21:03:30.398375 16424 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 21:03:30.427783 16888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:30.412563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48082 (local address 127.0.97.3:36811)
0929 21:03:30.419718 (+ 7155us) server_negotiation.cc:206] Beginning negotiation
0929 21:03:30.419726 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:03:30.419757 (+ 31us) server_negotiation.cc:407] Connection header received
0929 21:03:30.419847 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:03:30.419853 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:03:30.420122 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:03:30.420319 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:03:30.420878 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:30.422354 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:30.424736 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:30.426231 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:30.426401 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:30.426682 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0929 21:03:30.427179 (+ 497us) server_negotiation.cc:299] Negotiation successful
0929 21:03:30.427399 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6971,"thread_start_us":6704,"threads_started":1}
I20260929 21:03:30.427749 16885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:30.411788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36811 (local address 127.0.0.1:48082)
0929 21:03:30.419215 (+ 7427us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:30.419236 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 21:03:30.419358 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:03:30.420322 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:03:30.420326 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:03:30.420350 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:03:30.420668 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:03:30.420679 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:03:30.422492 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:03:30.422498 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:03:30.424573 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:03:30.424586 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:03:30.426822 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:03:30.426827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:03:30.426988 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:30.427017 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 21:03:30.427096 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 21:03:30.427197 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7293,"mutex_wait_us":181,"thread_start_us":1124,"threads_started":1}
I20260929 21:03:30.427821 16883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:30.409859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33721 (local address 127.0.0.1:39078)
0929 21:03:30.411267 (+ 1408us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:30.411313 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 21:03:30.411505 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:03:30.419220 (+ 7715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:03:30.419231 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:03:30.419274 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:03:30.419780 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:03:30.419799 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:03:30.422117 (+ 2318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:03:30.422127 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:03:30.424722 (+ 2595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:03:30.424737 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:03:30.426558 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:03:30.426565 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:03:30.426779 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:30.426855 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 21:03:30.426986 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 21:03:30.427190 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1115,"mutex_wait_us":130,"thread_start_us":946,"threads_started":1}
I20260929 21:03:30.428128 16887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:30.410282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46765 (local address 127.0.0.1:54826)
0929 21:03:30.412364 (+ 2082us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:30.412385 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 21:03:30.412504 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:03:30.419534 (+ 7030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:03:30.419538 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:03:30.419560 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:03:30.419932 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:03:30.419942 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:03:30.421823 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:03:30.421828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:03:30.424722 (+ 2894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:03:30.424737 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:03:30.426458 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:03:30.426464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:03:30.427356 (+ 892us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:30.427398 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 21:03:30.427490 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 21:03:30.427629 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1942,"mutex_wait_us":788,"thread_start_us":8662,"threads_started":1}
I20260929 21:03:30.428702 16884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:30.409919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39078 (local address 127.0.97.2:33721)
0929 21:03:30.418446 (+ 8527us) server_negotiation.cc:206] Beginning negotiation
0929 21:03:30.418467 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 21:03:30.418501 (+ 34us) server_negotiation.cc:407] Connection header received
0929 21:03:30.418619 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:03:30.418629 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:03:30.418919 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:03:30.419844 (+ 925us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:03:30.420094 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:30.421986 (+ 1892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:30.424908 (+ 2922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:30.426313 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:30.427857 (+ 1544us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:30.428079 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 21:03:30.428157 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 21:03:30.428344 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8312,"thread_start_us":8185,"threads_started":1}
I20260929 21:03:30.429921 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:30.410674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54826 (local address 127.0.97.1:46765)
0929 21:03:30.418055 (+ 7381us) server_negotiation.cc:206] Beginning negotiation
0929 21:03:30.418068 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 21:03:30.418103 (+ 35us) server_negotiation.cc:407] Connection header received
0929 21:03:30.418220 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:03:30.418228 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:03:30.418548 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:03:30.419563 (+ 1015us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:03:30.420095 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:30.421690 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:30.424938 (+ 3248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:30.426185 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:30.426718 (+ 533us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:30.426863 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 21:03:30.428401 (+ 1538us) server_negotiation.cc:299] Negotiation successful
0929 21:03:30.428583 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7154,"thread_start_us":7031,"threads_started":1}
I20260929 21:03:30.432408 16677 tablet_service.cc:1467] Processing CreateTablet for tablet 1347924bf4a045ed9f7a26dcd5edfb55 (DEFAULT_TABLE table=test-table [id=471c8be7115b432d80a3534ea3cfb28e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:03:30.432544 16808 tablet_service.cc:1467] Processing CreateTablet for tablet 1347924bf4a045ed9f7a26dcd5edfb55 (DEFAULT_TABLE table=test-table [id=471c8be7115b432d80a3534ea3cfb28e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:03:30.433969 16808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1347924bf4a045ed9f7a26dcd5edfb55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:03:30.434213 16677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1347924bf4a045ed9f7a26dcd5edfb55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:03:30.439517 16545 tablet_service.cc:1467] Processing CreateTablet for tablet 1347924bf4a045ed9f7a26dcd5edfb55 (DEFAULT_TABLE table=test-table [id=471c8be7115b432d80a3534ea3cfb28e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:03:30.441045 16545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1347924bf4a045ed9f7a26dcd5edfb55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:03:30.445829 16889 tablet_bootstrap.cc:492] T 1347924bf4a045ed9f7a26dcd5edfb55 P 3f66c38d69d34c9194b4d1264eddeb3d: Bootstrap starting.
I20260929 21:03:30.449529 16890 tablet_bootstrap.cc:492] T 1347924bf4a045ed9f7a26dcd5edfb55 P e5726fa404854fc4ba03654f587816d8: Bootstrap starting.
I20260929 21:03:30.450634 16889 tablet_bootstrap.cc:654] T 1347924bf4a045ed9f7a26dcd5edfb55 P 3f66c38d69d34c9194b4d1264eddeb3d: Neither blocks nor log segments found. Creating new log.
I20260929 21:03:30.451989 16889 log.cc:824] T 1347924bf4a045ed9f7a26dcd5edfb55 P 3f66c38d69d34c9194b4d1264eddeb3d: Log is configured to *not* fsync() on all Append() calls
I20260929 21:03:30.453039 16890 tablet_bootstrap.cc:654] T 1347924bf4a045ed9f7a26dcd5edfb55 P e5726fa404854fc4ba03654f587816d8: Neither blocks nor log segments found. Creating new log.
I20260929 21:03:30.454264 16890 log.cc:824] T 1347924bf4a045ed9f7a26dcd5edfb55 P e5726fa404854fc4ba03654f587816d8: Log is configured to *not* fsync() on all Append() calls
I20260929 21:03:30.455255 16889 tablet_bootstrap.cc:492] T 1347924bf4a045ed9f7a26dcd5edfb55 P 3f66c38d69d34c9194b4d1264eddeb3d: No bootstrap required, opened a new log
I20260929 21:03:30.455526 16889 ts_tablet_manager.cc:1397] T 1347924bf4a045ed9f7a26dcd5edfb55 P 3f66c38d69d34c9194b4d1264eddeb3d: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260929 21:03:30.456342 16893 tablet_bootstrap.cc:492] T 1347924bf4a045ed9f7a26dcd5edfb55 P 746b2763ccbf45d5874eb98263f37a3f: Bootstrap starting.
I20260929 21:03:30.457484 16890 tablet_bootstrap.cc:492] T 1347924bf4a045ed9f7a26dcd5edfb55 P e5726fa404854fc4ba03654f587816d8: No bootstrap required, opened a new log
I20260929 21:03:30.457769 16890 ts_tablet_manager.cc:1397] T 1347924bf4a045ed9f7a26dcd5edfb55 P e5726fa404854fc4ba03654f587816d8: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.006s
I20260929 21:03:30.459319 16893 tablet_bootstrap.cc:654] T 1347924bf4a045ed9f7a26dcd5edfb55 P 746b2763ccbf45d5874eb98263f37a3f: Neither blocks nor log segments found. Creating new log.
I20260929 21:03:30.460528 16893 log.cc:824] T 1347924bf4a045ed9f7a26dcd5edfb55 P 746b2763ccbf45d5874eb98263f37a3f: Log is configured to *not* fsync() on all Append() calls
I20260929 21:03:30.463030 16893 tablet_bootstrap.cc:492] T 1347924bf4a045ed9f7a26dcd5edfb55 P 746b2763ccbf45d5874eb98263f37a3f: No bootstrap required, opened a new log
I20260929 21:03:30.463300 16893 ts_tablet_manager.cc:1397] T 1347924bf4a045ed9f7a26dcd5edfb55 P 746b2763ccbf45d5874eb98263f37a3f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 21:03:30.467135 16889 raft_consensus.cc:348] T 1347924bf4a045ed9f7a26dcd5edfb55 P 3f66c38d69d34c9194b4d1264eddeb3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5726fa404854fc4ba03654f587816d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33721 } } peers { permanent_uuid: "746b2763ccbf45d5874eb98263f37a3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46765 } } peers { permanent_uuid: "3f66c38d69d34c9194b4d1264eddeb3d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36811 } }
I20260929 21:03:30.467595 16889 raft_consensus.cc:374] T 1347924bf4a045ed9f7a26dcd5edfb55 P 3f66c38d69d34c9194b4d1264eddeb3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:03:30.467697 16889 raft_consensus.cc:729] T 1347924bf4a045ed9f7a26dcd5edfb55 P 3f66c38d69d34c9194b4d1264eddeb3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f66c38d69d34c9194b4d1264eddeb3d, State: Initialized, Role: FOLLOWER
I20260929 21:03:30.468365 16889 consensus_queue.cc:260] T 1347924bf4a045ed9f7a26dcd5edfb55 P 3f66c38d69d34c9194b4d1264eddeb3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5726fa404854fc4ba03654f587816d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33721 } } peers { permanent_uuid: "746b2763ccbf45d5874eb98263f37a3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46765 } } peers { permanent_uuid: "3f66c38d69d34c9194b4d1264eddeb3d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36811 } }
I20260929 21:03:30.468459 16890 raft_consensus.cc:348] T 1347924bf4a045ed9f7a26dcd5edfb55 P e5726fa404854fc4ba03654f587816d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5726fa404854fc4ba03654f587816d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33721 } } peers { permanent_uuid: "746b2763ccbf45d5874eb98263f37a3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46765 } } peers { permanent_uuid: "3f66c38d69d34c9194b4d1264eddeb3d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36811 } }
I20260929 21:03:30.468895 16890 raft_consensus.cc:374] T 1347924bf4a045ed9f7a26dcd5edfb55 P e5726fa404854fc4ba03654f587816d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:03:30.469019 16890 raft_consensus.cc:729] T 1347924bf4a045ed9f7a26dcd5edfb55 P e5726fa404854fc4ba03654f587816d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5726fa404854fc4ba03654f587816d8, State: Initialized, Role: FOLLOWER
I20260929 21:03:30.469669 16873 heartbeater.cc:503] Master 127.0.97.62:46807 was elected leader, sending a full tablet report...
I20260929 21:03:30.469707 16890 consensus_queue.cc:260] T 1347924bf4a045ed9f7a26dcd5edfb55 P e5726fa404854fc4ba03654f587816d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5726fa404854fc4ba03654f587816d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33721 } } peers { permanent_uuid: "746b2763ccbf45d5874eb98263f37a3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46765 } } peers { permanent_uuid: "3f66c38d69d34c9194b4d1264eddeb3d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36811 } }
I20260929 21:03:30.470429 16889 ts_tablet_manager.cc:1428] T 1347924bf4a045ed9f7a26dcd5edfb55 P 3f66c38d69d34c9194b4d1264eddeb3d: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260929 21:03:30.471144 16610 heartbeater.cc:503] Master 127.0.97.62:46807 was elected leader, sending a full tablet report...
I20260929 21:03:30.471220 16893 raft_consensus.cc:348] T 1347924bf4a045ed9f7a26dcd5edfb55 P 746b2763ccbf45d5874eb98263f37a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5726fa404854fc4ba03654f587816d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33721 } } peers { permanent_uuid: "746b2763ccbf45d5874eb98263f37a3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46765 } } peers { permanent_uuid: "3f66c38d69d34c9194b4d1264eddeb3d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36811 } }
I20260929 21:03:30.471660 16893 raft_consensus.cc:374] T 1347924bf4a045ed9f7a26dcd5edfb55 P 746b2763ccbf45d5874eb98263f37a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:03:30.471762 16893 raft_consensus.cc:729] T 1347924bf4a045ed9f7a26dcd5edfb55 P 746b2763ccbf45d5874eb98263f37a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 746b2763ccbf45d5874eb98263f37a3f, State: Initialized, Role: FOLLOWER
I20260929 21:03:30.472443 16893 consensus_queue.cc:260] T 1347924bf4a045ed9f7a26dcd5edfb55 P 746b2763ccbf45d5874eb98263f37a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5726fa404854fc4ba03654f587816d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33721 } } peers { permanent_uuid: "746b2763ccbf45d5874eb98263f37a3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46765 } } peers { permanent_uuid: "3f66c38d69d34c9194b4d1264eddeb3d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36811 } }
I20260929 21:03:30.473328 16890 ts_tablet_manager.cc:1428] T 1347924bf4a045ed9f7a26dcd5edfb55 P e5726fa404854fc4ba03654f587816d8: Time spent starting tablet: real 0.015s user 0.012s sys 0.003s
I20260929 21:03:30.474278 16742 heartbeater.cc:503] Master 127.0.97.62:46807 was elected leader, sending a full tablet report...
I20260929 21:03:30.476734 16893 ts_tablet_manager.cc:1428] T 1347924bf4a045ed9f7a26dcd5edfb55 P 746b2763ccbf45d5874eb98263f37a3f: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260929 21:03:30.546105 16896 raft_consensus.cc:482] T 1347924bf4a045ed9f7a26dcd5edfb55 P e5726fa404854fc4ba03654f587816d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 21:03:30.546398 16896 raft_consensus.cc:504] T 1347924bf4a045ed9f7a26dcd5edfb55 P e5726fa404854fc4ba03654f587816d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5726fa404854fc4ba03654f587816d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33721 } } peers { permanent_uuid: "746b2763ccbf45d5874eb98263f37a3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46765 } } peers { permanent_uuid: "3f66c38d69d34c9194b4d1264eddeb3d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36811 } }
I20260929 21:03:30.548074 16896 leader_election.cc:288] T 1347924bf4a045ed9f7a26dcd5edfb55 P e5726fa404854fc4ba03654f587816d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 746b2763ccbf45d5874eb98263f37a3f (127.0.97.1:46765), 3f66c38d69d34c9194b4d1264eddeb3d (127.0.97.3:36811)
I20260929 21:03:30.554720 16888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:30.548446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41153 (local address 127.0.97.3:36811)
0929 21:03:30.548680 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 21:03:30.548688 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:03:30.549478 (+ 790us) server_negotiation.cc:407] Connection header received
0929 21:03:30.549689 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:03:30.549693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:03:30.549750 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:03:30.549840 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:03:30.550467 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:30.551162 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:30.553015 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:30.553794 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:30.553941 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:30.554052 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 21:03:30.554331 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 21:03:30.554465 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 21:03:30.554809 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:30.548428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57995 (local address 127.0.97.1:46765)
0929 21:03:30.548638 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 21:03:30.548646 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:03:30.549616 (+ 970us) server_negotiation.cc:407] Connection header received
0929 21:03:30.549812 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:03:30.549818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:03:30.549886 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:03:30.549986 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:03:30.550597 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:30.551285 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:30.553096 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:30.553873 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:30.554032 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:30.554133 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 21:03:30.554431 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 21:03:30.554556 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 21:03:30.554893 16898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:30.548312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36811 (local address 127.0.97.2:41153)
0929 21:03:30.549361 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:30.549376 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 21:03:30.549542 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:03:30.549854 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:03:30.549858 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:03:30.549881 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:03:30.550310 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:03:30.550319 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:03:30.551287 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:03:30.551294 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:03:30.552810 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:03:30.552820 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:03:30.554015 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:03:30.554022 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:03:30.554175 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:30.554197 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 21:03:30.554305 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 21:03:30.554421 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":933,"mutex_wait_us":66,"spinlock_wait_cycles":12544,"thread_start_us":698,"threads_started":1}
I20260929 21:03:30.555301 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:30.548482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46765 (local address 127.0.97.2:57995)
0929 21:03:30.549504 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:30.549518 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 21:03:30.549666 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:03:30.550002 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:03:30.550006 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:03:30.550044 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:03:30.550445 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:03:30.550453 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:03:30.551425 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:03:30.551431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:03:30.552944 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:03:30.552956 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:03:30.554096 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:03:30.554102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:03:30.554269 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:30.554291 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 21:03:30.554396 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 21:03:30.554510 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":915,"mutex_wait_us":6,"thread_start_us":607,"threads_started":1}
I20260929 21:03:30.555872 16828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1347924bf4a045ed9f7a26dcd5edfb55" candidate_uuid: "e5726fa404854fc4ba03654f587816d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f66c38d69d34c9194b4d1264eddeb3d" is_pre_election: true
I20260929 21:03:30.556329 16565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1347924bf4a045ed9f7a26dcd5edfb55" candidate_uuid: "e5726fa404854fc4ba03654f587816d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "746b2763ccbf45d5874eb98263f37a3f" is_pre_election: true
I20260929 21:03:30.556464 16828 raft_consensus.cc:2454] T 1347924bf4a045ed9f7a26dcd5edfb55 P 3f66c38d69d34c9194b4d1264eddeb3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5726fa404854fc4ba03654f587816d8 in term 0.
I20260929 21:03:30.556927 16565 raft_consensus.cc:2454] T 1347924bf4a045ed9f7a26dcd5edfb55 P 746b2763ccbf45d5874eb98263f37a3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5726fa404854fc4ba03654f587816d8 in term 0.
I20260929 21:03:30.557189 16632 leader_election.cc:302] T 1347924bf4a045ed9f7a26dcd5edfb55 P e5726fa404854fc4ba03654f587816d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f66c38d69d34c9194b4d1264eddeb3d, e5726fa404854fc4ba03654f587816d8; no voters:
I20260929 21:03:30.557608 16896 raft_consensus.cc:2781] T 1347924bf4a045ed9f7a26dcd5edfb55 P e5726fa404854fc4ba03654f587816d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 21:03:30.557758 16896 raft_consensus.cc:482] T 1347924bf4a045ed9f7a26dcd5edfb55 P e5726fa404854fc4ba03654f587816d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 21:03:30.557866 16896 raft_consensus.cc:3037] T 1347924bf4a045ed9f7a26dcd5edfb55 P e5726fa404854fc4ba03654f587816d8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:03:30.559981 16896 raft_consensus.cc:504] T 1347924bf4a045ed9f7a26dcd5edfb55 P e5726fa404854fc4ba03654f587816d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5726fa404854fc4ba03654f587816d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33721 } } peers { permanent_uuid: "746b2763ccbf45d5874eb98263f37a3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46765 } } peers { permanent_uuid: "3f66c38d69d34c9194b4d1264eddeb3d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36811 } }
I20260929 21:03:30.560523 16896 leader_election.cc:288] T 1347924bf4a045ed9f7a26dcd5edfb55 P e5726fa404854fc4ba03654f587816d8 [CANDIDATE]: Term 1 election: Requested vote from peers 746b2763ccbf45d5874eb98263f37a3f (127.0.97.1:46765), 3f66c38d69d34c9194b4d1264eddeb3d (127.0.97.3:36811)
I20260929 21:03:30.560959 16828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1347924bf4a045ed9f7a26dcd5edfb55" candidate_uuid: "e5726fa404854fc4ba03654f587816d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f66c38d69d34c9194b4d1264eddeb3d"
I20260929 21:03:30.560937 16565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1347924bf4a045ed9f7a26dcd5edfb55" candidate_uuid: "e5726fa404854fc4ba03654f587816d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "746b2763ccbf45d5874eb98263f37a3f"
I20260929 21:03:30.561153 16828 raft_consensus.cc:3037] T 1347924bf4a045ed9f7a26dcd5edfb55 P 3f66c38d69d34c9194b4d1264eddeb3d [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:03:30.561149 16565 raft_consensus.cc:3037] T 1347924bf4a045ed9f7a26dcd5edfb55 P 746b2763ccbf45d5874eb98263f37a3f [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:03:30.564415 16828 raft_consensus.cc:2454] T 1347924bf4a045ed9f7a26dcd5edfb55 P 3f66c38d69d34c9194b4d1264eddeb3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5726fa404854fc4ba03654f587816d8 in term 1.
I20260929 21:03:30.564414 16565 raft_consensus.cc:2454] T 1347924bf4a045ed9f7a26dcd5edfb55 P 746b2763ccbf45d5874eb98263f37a3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5726fa404854fc4ba03654f587816d8 in term 1.
I20260929 21:03:30.564857 16632 leader_election.cc:302] T 1347924bf4a045ed9f7a26dcd5edfb55 P e5726fa404854fc4ba03654f587816d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f66c38d69d34c9194b4d1264eddeb3d, e5726fa404854fc4ba03654f587816d8; no voters:
I20260929 21:03:30.565124 16896 raft_consensus.cc:2781] T 1347924bf4a045ed9f7a26dcd5edfb55 P e5726fa404854fc4ba03654f587816d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:03:30.565587 16896 raft_consensus.cc:686] T 1347924bf4a045ed9f7a26dcd5edfb55 P e5726fa404854fc4ba03654f587816d8 [term 1 LEADER]: Becoming Leader. State: Replica: e5726fa404854fc4ba03654f587816d8, State: Running, Role: LEADER
I20260929 21:03:30.566468 16896 consensus_queue.cc:237] T 1347924bf4a045ed9f7a26dcd5edfb55 P e5726fa404854fc4ba03654f587816d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5726fa404854fc4ba03654f587816d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33721 } } peers { permanent_uuid: "746b2763ccbf45d5874eb98263f37a3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46765 } } peers { permanent_uuid: "3f66c38d69d34c9194b4d1264eddeb3d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36811 } }
I20260929 21:03:30.570801 16422 catalog_manager.cc:5495] T 1347924bf4a045ed9f7a26dcd5edfb55 P e5726fa404854fc4ba03654f587816d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to e5726fa404854fc4ba03654f587816d8 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "e5726fa404854fc4ba03654f587816d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5726fa404854fc4ba03654f587816d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "746b2763ccbf45d5874eb98263f37a3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f66c38d69d34c9194b4d1264eddeb3d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36811 } health_report { overall_health: UNKNOWN } } }
W20260929 21:03:30.591536 16874 tablet.cc:2369] T 1347924bf4a045ed9f7a26dcd5edfb55 P 3f66c38d69d34c9194b4d1264eddeb3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 21:03:30.612351 16884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:30.607365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39090 (local address 127.0.97.2:33721)
0929 21:03:30.607551 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 21:03:30.607556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:03:30.607571 (+ 15us) server_negotiation.cc:407] Connection header received
0929 21:03:30.607657 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:03:30.607660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:03:30.607742 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:03:30.607862 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:03:30.608172 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:30.609035 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:30.610395 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:30.610785 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:30.610874 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:30.611285 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:03:30.611832 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:03:30.611940 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 21:03:30.612029 (+ 89us) server_negotiation.cc:299] Negotiation successful
0929 21:03:30.612092 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 21:03:30.620173 16828 raft_consensus.cc:1261] T 1347924bf4a045ed9f7a26dcd5edfb55 P 3f66c38d69d34c9194b4d1264eddeb3d [term 1 FOLLOWER]: Refusing update from remote peer e5726fa404854fc4ba03654f587816d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:03:30.620173 16565 raft_consensus.cc:1261] T 1347924bf4a045ed9f7a26dcd5edfb55 P 746b2763ccbf45d5874eb98263f37a3f [term 1 FOLLOWER]: Refusing update from remote peer e5726fa404854fc4ba03654f587816d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:03:30.621152 16900 consensus_queue.cc:1035] T 1347924bf4a045ed9f7a26dcd5edfb55 P e5726fa404854fc4ba03654f587816d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f66c38d69d34c9194b4d1264eddeb3d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:03:30.621500 16896 consensus_queue.cc:1035] T 1347924bf4a045ed9f7a26dcd5edfb55 P e5726fa404854fc4ba03654f587816d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "746b2763ccbf45d5874eb98263f37a3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:03:30.650585 16422 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 21:03:30.650842 16422 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 21:03:30.654073 16422 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21: Sending DeleteTablet for 3 replicas of tablet 1347924bf4a045ed9f7a26dcd5edfb55
I20260929 21:03:30.655196 16545 tablet_service.cc:1514] Processing DeleteTablet for tablet 1347924bf4a045ed9f7a26dcd5edfb55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:03:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54826
I20260929 21:03:30.655309 16677 tablet_service.cc:1514] Processing DeleteTablet for tablet 1347924bf4a045ed9f7a26dcd5edfb55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:03:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39078
I20260929 21:03:30.655640 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 16461
I20260929 21:03:30.655941 16808 tablet_service.cc:1514] Processing DeleteTablet for tablet 1347924bf4a045ed9f7a26dcd5edfb55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:03:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48082
I20260929 21:03:30.656546 16912 tablet_replica.cc:330] T 1347924bf4a045ed9f7a26dcd5edfb55 P 746b2763ccbf45d5874eb98263f37a3f: stopping tablet replica
I20260929 21:03:30.656973 16912 raft_consensus.cc:2229] T 1347924bf4a045ed9f7a26dcd5edfb55 P 746b2763ccbf45d5874eb98263f37a3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:03:30.657475 16912 raft_consensus.cc:2258] T 1347924bf4a045ed9f7a26dcd5edfb55 P 746b2763ccbf45d5874eb98263f37a3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:03:30.658020 16914 tablet_replica.cc:330] T 1347924bf4a045ed9f7a26dcd5edfb55 P 3f66c38d69d34c9194b4d1264eddeb3d: stopping tablet replica
I20260929 21:03:30.658224 16913 tablet_replica.cc:330] T 1347924bf4a045ed9f7a26dcd5edfb55 P e5726fa404854fc4ba03654f587816d8: stopping tablet replica
I20260929 21:03:30.658553 16913 raft_consensus.cc:2229] T 1347924bf4a045ed9f7a26dcd5edfb55 P e5726fa404854fc4ba03654f587816d8 [term 1 LEADER]: Raft consensus shutting down.
I20260929 21:03:30.658526 16914 raft_consensus.cc:2229] T 1347924bf4a045ed9f7a26dcd5edfb55 P 3f66c38d69d34c9194b4d1264eddeb3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:03:30.659224 16913 raft_consensus.cc:2258] T 1347924bf4a045ed9f7a26dcd5edfb55 P e5726fa404854fc4ba03654f587816d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:03:30.660567 16914 raft_consensus.cc:2258] T 1347924bf4a045ed9f7a26dcd5edfb55 P 3f66c38d69d34c9194b4d1264eddeb3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:03:30.661670 16912 ts_tablet_manager.cc:1905] T 1347924bf4a045ed9f7a26dcd5edfb55 P 746b2763ccbf45d5874eb98263f37a3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:03:30.661640 16913 ts_tablet_manager.cc:1905] T 1347924bf4a045ed9f7a26dcd5edfb55 P e5726fa404854fc4ba03654f587816d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:03:30.666880 16914 ts_tablet_manager.cc:1905] T 1347924bf4a045ed9f7a26dcd5edfb55 P 3f66c38d69d34c9194b4d1264eddeb3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:03:30.669023 16913 ts_tablet_manager.cc:1918] T 1347924bf4a045ed9f7a26dcd5edfb55 P e5726fa404854fc4ba03654f587816d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:03:30.669181 16913 log.cc:1196] T 1347924bf4a045ed9f7a26dcd5edfb55 P e5726fa404854fc4ba03654f587816d8: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/wal/wals/1347924bf4a045ed9f7a26dcd5edfb55
I20260929 21:03:30.669720 16913 ts_tablet_manager.cc:1939] T 1347924bf4a045ed9f7a26dcd5edfb55 P e5726fa404854fc4ba03654f587816d8: Deleting consensus metadata
I20260929 21:03:30.671516 16410 catalog_manager.cc:4841] TS e5726fa404854fc4ba03654f587816d8 (127.0.97.2:33721): tablet 1347924bf4a045ed9f7a26dcd5edfb55 (table test-table [id=471c8be7115b432d80a3534ea3cfb28e]) successfully deleted
I20260929 21:03:30.673758 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:30.656714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54838 (local address 127.0.97.1:46765)
0929 21:03:30.656896 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 21:03:30.656902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:03:30.656929 (+ 27us) server_negotiation.cc:407] Connection header received
0929 21:03:30.656988 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:03:30.656996 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:03:30.657084 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:03:30.657202 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:03:30.658982 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:30.660141 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:30.662258 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:30.662823 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:30.662964 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:30.666316 (+ 3352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:03:30.666327 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:03:30.666344 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:03:30.666384 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:03:30.670242 (+ 3858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:03:30.671035 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:03:30.671042 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:03:30.671046 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:03:30.671192 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:03:30.671576 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:03:30.671580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:03:30.671583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:03:30.672274 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:03:30.672577 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0929 21:03:30.672976 (+ 399us) server_negotiation.cc:299] Negotiation successful
0929 21:03:30.673401 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 21:03:30.674505 16912 ts_tablet_manager.cc:1918] T 1347924bf4a045ed9f7a26dcd5edfb55 P 746b2763ccbf45d5874eb98263f37a3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:03:30.674650 16605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:03:30.674669 16912 log.cc:1196] T 1347924bf4a045ed9f7a26dcd5edfb55 P 746b2763ccbf45d5874eb98263f37a3f: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/wal/wals/1347924bf4a045ed9f7a26dcd5edfb55
I20260929 21:03:30.676362 16914 ts_tablet_manager.cc:1918] T 1347924bf4a045ed9f7a26dcd5edfb55 P 3f66c38d69d34c9194b4d1264eddeb3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:03:30.676535 16914 log.cc:1196] T 1347924bf4a045ed9f7a26dcd5edfb55 P 3f66c38d69d34c9194b4d1264eddeb3d: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/wal/wals/1347924bf4a045ed9f7a26dcd5edfb55
I20260929 21:03:30.677084 16914 ts_tablet_manager.cc:1939] T 1347924bf4a045ed9f7a26dcd5edfb55 P 3f66c38d69d34c9194b4d1264eddeb3d: Deleting consensus metadata
I20260929 21:03:30.678860 16410 catalog_manager.cc:4841] TS 3f66c38d69d34c9194b4d1264eddeb3d (127.0.97.3:36811): tablet 1347924bf4a045ed9f7a26dcd5edfb55 (table test-table [id=471c8be7115b432d80a3534ea3cfb28e]) successfully deleted
I20260929 21:03:30.791008 16912 ts_tablet_manager.cc:1939] T 1347924bf4a045ed9f7a26dcd5edfb55 P 746b2763ccbf45d5874eb98263f37a3f: Deleting consensus metadata
I20260929 21:03:30.792630 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 16461
I20260929 21:03:30.792680 16409 catalog_manager.cc:4841] TS 746b2763ccbf45d5874eb98263f37a3f (127.0.97.1:46765): tablet 1347924bf4a045ed9f7a26dcd5edfb55 (table test-table [id=471c8be7115b432d80a3534ea3cfb28e]) successfully deleted
I20260929 21:03:30.810356 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 16614
I20260929 21:03:30.825080 16884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:30.811162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39102 (local address 127.0.97.2:33721)
0929 21:03:30.811348 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 21:03:30.811354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:03:30.811378 (+ 24us) server_negotiation.cc:407] Connection header received
0929 21:03:30.811557 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:03:30.811563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:03:30.811648 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:03:30.811763 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:03:30.813428 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:30.814285 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:30.815586 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:30.816019 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:30.816152 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:30.818730 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:03:30.818741 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:03:30.818758 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:03:30.818799 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:03:30.821972 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:03:30.822674 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:03:30.822687 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:03:30.822692 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:03:30.822846 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:03:30.823316 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:03:30.823321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:03:30.823324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:03:30.823665 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:03:30.824004 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0929 21:03:30.824335 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 21:03:30.824574 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 21:03:30.825783 16737 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:03:30.938766 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 16614
I20260929 21:03:30.958112 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 16745
I20260929 21:03:30.971393 16888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:30.958907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48090 (local address 127.0.97.3:36811)
0929 21:03:30.959093 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 21:03:30.959100 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:03:30.959119 (+ 19us) server_negotiation.cc:407] Connection header received
0929 21:03:30.959316 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:03:30.959321 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:03:30.959394 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:03:30.959499 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:03:30.960892 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:30.961737 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:30.963045 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:03:30.963449 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:03:30.963579 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:03:30.966180 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:03:30.966196 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:03:30.966212 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:03:30.966260 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:03:30.968659 (+ 2399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:03:30.969390 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:03:30.969400 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:03:30.969403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:03:30.969543 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:03:30.969897 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:03:30.969903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:03:30.969906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:03:30.970259 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:03:30.970552 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0929 21:03:30.970837 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 21:03:30.971091 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 21:03:30.972046 16868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:03:31.060470 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 16745
I20260929 21:03:31.077013 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 16391
I20260929 21:03:31.078168 16454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:03:31.194685 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 16391
I20260929 21:03:51.217823 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46807
--webserver_interface=127.0.97.62
--webserver_port=43369
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:03:51.545635 16919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:03:51.545966 16919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:03:51.546135 16919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:03:51.553522 16919 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 21:03:51.553647 16919 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 21:03:51.553738 16919 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 21:03:51.553833 16919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:03:51.553929 16919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:03:51.554047 16919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:03:51.554134 16919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:03:51.554212 16919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:03:51.564129 16919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=43369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:03:51.565531 16919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:03:51.566883 16919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:03:51.579196 16925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:03:51.579317 16924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:03:51.579293 16927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:03:51.579955 16919 server_base.cc:1034] running on GCE node
I20260929 21:03:51.580822 16919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:03:51.582346 16919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:03:51.583600 16919 hybrid_clock.cc:648] HybridClock initialized: now 1790715831583537 us; error 40 us; skew 500 ppm
I20260929 21:03:51.583951 16919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:03:51 dist-test-slave-1dms krb5kdc[15840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:03:51.594413 16919 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 21:03:51.596026 16919 webserver.cc:460] Webserver started at http://127.0.97.62:43369/ using document root <none> and password file <none>
I20260929 21:03:51.596746 16919 fs_manager.cc:359] Metadata directory not provided
I20260929 21:03:51.596902 16919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:03:51.602293 16919 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260929 21:03:51.605026 16934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:03:51.606279 16919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 21:03:51.606477 16919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "885b912376d24bc48fda185253979e21"
format_stamp: "Formatted at 2026-09-29 21:03:05 on dist-test-slave-1dms"
server_key: "516751997f5064282644b29234ab7c4e"
server_key_iv: "649df24baeee8faf33fbf6fc42ba08b4"
server_key_version: "encryptionkey@0"
I20260929 21:03:51.607404 16919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:03:51.633935 16919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:03:51.638310 16919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:03:51.638621 16919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:03:51.657485 16919 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46807
I20260929 21:03:51.657519 16985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46807 every 8 connection(s)
I20260929 21:03:51.658967 16919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:03:51.661130 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 16919
I20260929 21:03:51.662222 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:46765
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46807
--webserver_port=33381
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 21:03:51.669648 16986 sys_catalog.cc:263] Verifying existing consensus state
I20260929 21:03:51.673926 16986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21: Bootstrap starting.
I20260929 21:03:51.695248 16986 log.cc:824] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21: Log is configured to *not* fsync() on all Append() calls
I20260929 21:03:51.710633 16986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 21:03:51.711266 16986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21: Bootstrap complete.
I20260929 21:03:51.720191 16986 raft_consensus.cc:348] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885b912376d24bc48fda185253979e21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46807 } }
I20260929 21:03:51.721081 16986 raft_consensus.cc:729] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 885b912376d24bc48fda185253979e21, State: Initialized, Role: FOLLOWER
I20260929 21:03:51.721660 16986 consensus_queue.cc:260] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885b912376d24bc48fda185253979e21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46807 } }
I20260929 21:03:51.721875 16986 raft_consensus.cc:388] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:03:51.722018 16986 raft_consensus.cc:482] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:03:51.722195 16986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [term 2 FOLLOWER]: Advancing to term 3
I20260929 21:03:51.726341 16986 raft_consensus.cc:504] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885b912376d24bc48fda185253979e21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46807 } }
I20260929 21:03:51.726814 16986 leader_election.cc:302] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 885b912376d24bc48fda185253979e21; no voters:
I20260929 21:03:51.727485 16986 leader_election.cc:288] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 21:03:51.727620 16990 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 21:03:51.728365 16990 raft_consensus.cc:686] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [term 3 LEADER]: Becoming Leader. State: Replica: 885b912376d24bc48fda185253979e21, State: Running, Role: LEADER
I20260929 21:03:51.729005 16990 consensus_queue.cc:237] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885b912376d24bc48fda185253979e21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46807 } }
I20260929 21:03:51.729553 16986 sys_catalog.cc:564] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:03:51.730643 16990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 885b912376d24bc48fda185253979e21. Latest consensus state: current_term: 3 leader_uuid: "885b912376d24bc48fda185253979e21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885b912376d24bc48fda185253979e21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46807 } } }
I20260929 21:03:51.730902 16990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [sys.catalog]: This master's current role is: LEADER
I20260929 21:03:51.731954 16994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:03:51.731679 16992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "885b912376d24bc48fda185253979e21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885b912376d24bc48fda185253979e21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46807 } } }
I20260929 21:03:51.732429 16992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21 [sys.catalog]: This master's current role is: LEADER
I20260929 21:03:51.754060 16994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:03:51.755990 16994 catalog_manager.cc:1259] Loaded cluster ID: 0d536ebc009246f2ae6910666b207a89
I20260929 21:03:51.756213 16994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:03:51.761418 16994 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:03:51.763696 16994 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21: Loaded TSK: 0
I20260929 21:03:51.763957 16994 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 885b912376d24bc48fda185253979e21: Loaded TSK: 1
F20260929 21:03:51.765440 16994 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f90430f49ad google::LogMessage::Fail() at ??:0
@ 0x7f90430f788a google::LogMessage::SendToLog() at ??:0
@ 0x7f90430f44de google::LogMessage::Flush() at ??:0
@ 0x7f90430f5a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9046e587d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f9046e4773e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f9058b231c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f9058b22783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f9058bae2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f9058bae0c4 std::_Function_handler<>::_M_invoke() at ??:0
I20260929 21:03:51.811687 17009 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f9058bf85e9 std::function<>::operator()() at ??:0
@ 0x7f9058b26cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f9058b24c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9058bac610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f9058bac439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f9045d40345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9045d4391c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f9045d43799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f9045d1eeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9048ff86db start_thread at ??:0
@ 0x7f9041f8e61f clone at ??:0
W20260929 21:03:52.061336 16988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:03:52.061605 16988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:03:52.061744 16988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:03:52.069061 16988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:03:52.069164 16988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:03:52.069370 16988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 21:03:52.079218 16988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:46765
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=33381
--tserver_master_addrs=127.0.97.62:46807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:03:52.080514 16988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:03:52.081991 16988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:03:52.094823 17033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:03:52.094806 17034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:03:52.095064 17036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:03:52.095562 16988 server_base.cc:1034] running on GCE node
I20260929 21:03:52.096184 16988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:03:52.097182 16988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:03:52.098415 16988 hybrid_clock.cc:648] HybridClock initialized: now 1790715832098349 us; error 58 us; skew 500 ppm
I20260929 21:03:52.098742 16988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:03:52 dist-test-slave-1dms krb5kdc[15840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:03:52.101890 16988 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 21:03:52.103257 16988 webserver.cc:460] Webserver started at http://127.0.97.1:33381/ using document root <none> and password file <none>
I20260929 21:03:52.103893 16988 fs_manager.cc:359] Metadata directory not provided
I20260929 21:03:52.103999 16988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:03:52.109175 16988 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 21:03:52.111727 17043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:03:52.113157 16988 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:03:52.113289 16988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/wal
uuid: "746b2763ccbf45d5874eb98263f37a3f"
format_stamp: "Formatted at 2026-09-29 21:03:06 on dist-test-slave-1dms"
server_key: "34cd18bd130d4157d17121a84ce76ee5"
server_key_iv: "cf2f5f75793e972df16db7d45331bc91"
server_key_version: "encryptionkey@0"
I20260929 21:03:52.114280 16988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:03:52.135238 16988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:03:52.139375 16988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:03:52.139590 16988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:03:52.140473 16988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:03:52.141731 16988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:03:52.141814 16988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:03:52.141896 16988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:03:52.141955 16988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:03:52.176146 16988 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46765
I20260929 21:03:52.176237 17155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46765 every 8 connection(s)
I20260929 21:03:52.177657 16988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
I20260929 21:03:52.179018 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 16988
I20260929 21:03:52.179816 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:33721
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46807
--webserver_port=37831
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 21:03:52.182900 17158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:52.180987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.1:35895)
0929 21:03:52.181806 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:52.182052 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":312,"threads_started":1}
W20260929 21:03:52.183578 17156 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46807 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:46807: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
I20260929 21:03:52.184408 17158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:52.184082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.1:57581)
0929 21:03:52.184232 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:52.184306 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 21:03:52.185292 17158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:52.184975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.1:53773)
0929 21:03:52.185106 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:52.185189 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 21:03:52.185514 17156 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 21:03:52.500114 17159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:03:52.500433 17159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:03:52.500619 17159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:03:52.512943 17159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:03:52.513077 17159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:03:52.513263 17159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 21:03:52.524591 17159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:33721
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=37831
--tserver_master_addrs=127.0.97.62:46807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:03:52.525866 17159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:03:52.527172 17159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:03:52.539589 17167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:03:52.539613 17164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:03:52.539739 17165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:03:52.540360 17159 server_base.cc:1034] running on GCE node
I20260929 21:03:52.541002 17159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:03:52.541867 17159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:03:52.543074 17159 hybrid_clock.cc:648] HybridClock initialized: now 1790715832543035 us; error 24 us; skew 500 ppm
I20260929 21:03:52.543421 17159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:03:52 dist-test-slave-1dms krb5kdc[15840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:03:52.546742 17159 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 21:03:52.548149 17159 webserver.cc:460] Webserver started at http://127.0.97.2:37831/ using document root <none> and password file <none>
I20260929 21:03:52.548889 17159 fs_manager.cc:359] Metadata directory not provided
I20260929 21:03:52.549042 17159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:03:52.554250 17159 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260929 21:03:52.556910 17174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:03:52.558212 17159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 21:03:52.558387 17159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/wal
uuid: "e5726fa404854fc4ba03654f587816d8"
format_stamp: "Formatted at 2026-09-29 21:03:07 on dist-test-slave-1dms"
server_key: "7cad1c146ac263cca6b5a42bf890ae28"
server_key_iv: "42dc0d4024cce989637d4b869183890f"
server_key_version: "encryptionkey@0"
I20260929 21:03:52.559367 17159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:03:52.575563 17159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:03:52.579587 17159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:03:52.579862 17159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:03:52.580889 17159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:03:52.582217 17159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:03:52.582341 17159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:03:52.582481 17159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:03:52.582576 17159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:03:52.616231 17159 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33721
I20260929 21:03:52.616317 17286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33721 every 8 connection(s)
I20260929 21:03:52.617956 17159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
I20260929 21:03:52.622771 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:52.620999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.2:45211)
0929 21:03:52.621716 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:52.621907 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":276,"threads_started":1}
W20260929 21:03:52.623502 17287 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46807 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:46807: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
I20260929 21:03:52.624270 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:52.623974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.2:35235)
0929 21:03:52.624108 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:52.624172 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 21:03:52.625103 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 17159
I20260929 21:03:52.625159 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:52.624810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.2:41511)
0929 21:03:52.624966 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:52.625056 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260929 21:03:52.625402 17287 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 21:03:52.625860 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:36811
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46807
--webserver_port=36963
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:03:52.942134 17290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:03:52.942447 17290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:03:52.942622 17290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:03:52.949949 17290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:03:52.950096 17290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:03:52.950265 17290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 21:03:52.959905 17290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:36811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=36963
--tserver_master_addrs=127.0.97.62:46807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:03:52.961306 17290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:03:52.962596 17290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:03:52.977415 17295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:03:52.977375 17296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:03:52.977352 17298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:03:52.978168 17290 server_base.cc:1034] running on GCE node
I20260929 21:03:52.978892 17290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:03:52.979840 17290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:03:52.981078 17290 hybrid_clock.cc:648] HybridClock initialized: now 1790715832981030 us; error 56 us; skew 500 ppm
I20260929 21:03:52.981406 17290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:03:52 dist-test-slave-1dms krb5kdc[15840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:03:52.984812 17290 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 21:03:52.986279 17290 webserver.cc:460] Webserver started at http://127.0.97.3:36963/ using document root <none> and password file <none>
I20260929 21:03:52.986969 17290 fs_manager.cc:359] Metadata directory not provided
I20260929 21:03:52.987115 17290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:03:52.992280 17290 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 21:03:52.994871 17305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:03:52.996088 17290 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 21:03:52.996263 17290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/wal
uuid: "3f66c38d69d34c9194b4d1264eddeb3d"
format_stamp: "Formatted at 2026-09-29 21:03:07 on dist-test-slave-1dms"
server_key: "b0facff35d5cf4a30c4a791c4c936e1d"
server_key_iv: "3944b588241ddd381526015d31495ff4"
server_key_version: "encryptionkey@0"
I20260929 21:03:52.997285 17290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:03:53.034127 17290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:03:53.038043 17290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:03:53.038314 17290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:03:53.039289 17290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:03:53.040591 17290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:03:53.040714 17290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:03:53.040863 17290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:03:53.040961 17290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260929 21:03:53.074019 17290 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36811
I20260929 21:03:53.074069 17417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36811 every 8 connection(s)
I20260929 21:03:53.075469 17290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
I20260929 21:03:53.077823 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 17290
I20260929 21:03:53.080385 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:53.078546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.3:52531)
0929 21:03:53.079345 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:53.079547 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":333,"threads_started":1}
W20260929 21:03:53.081197 17418 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46807 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:46807: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
I20260929 21:03:53.081991 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:53.081670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.3:54443)
0929 21:03:53.081818 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:53.081894 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 21:03:53.082816 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:53.082529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.3:56833)
0929 21:03:53.082641 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:53.082709 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 21:03:53.083096 17418 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 21:03:53.186733 17422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:53.185974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.1:60695)
0929 21:03:53.186430 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:53.186517 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":199,"threads_started":1}
I20260929 21:03:53.626806 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:53.625984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.2:52389)
0929 21:03:53.626486 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:53.626577 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":232,"threads_started":1}
I20260929 21:03:54.084415 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:54.083577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.3:55485)
0929 21:03:54.084026 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:54.084110 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":220,"threads_started":1}
I20260929 21:03:54.188195 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:54.187532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.1:52925)
0929 21:03:54.187950 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:54.188022 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":206,"threads_started":1}
I20260929 21:03:54.628358 17426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:54.627679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.2:38899)
0929 21:03:54.628119 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:54.628203 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":225,"threads_started":1}
I20260929 21:03:55.086004 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:55.085334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.3:49567)
0929 21:03:55.085757 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:55.085840 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":220,"threads_started":1}
I20260929 21:03:55.189729 17428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:55.189039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.1:52963)
0929 21:03:55.189485 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:55.189563 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":235,"threads_started":1}
I20260929 21:03:55.629842 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:55.629181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.2:50385)
0929 21:03:55.629601 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:55.629677 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":215,"threads_started":1}
I20260929 21:03:56.087582 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:56.086827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.3:40961)
0929 21:03:56.087333 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:56.087409 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":261,"threads_started":1}
I20260929 21:03:56.191231 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:56.190537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.1:60591)
0929 21:03:56.190955 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:56.191040 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":212,"threads_started":1}
I20260929 21:03:56.631419 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:56.630733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.2:56591)
0929 21:03:56.631150 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:56.631228 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":202,"threads_started":1}
I20260929 21:03:57.089110 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:57.088376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.3:43093)
0929 21:03:57.088796 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:57.088937 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":224,"threads_started":1}
I20260929 21:03:57.192773 17434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:57.192086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.1:51477)
0929 21:03:57.192530 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:57.192607 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":228,"threads_started":1}
I20260929 21:03:57.633009 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:57.632304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.2:59845)
0929 21:03:57.632752 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:57.632828 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":238,"threads_started":1}
I20260929 21:03:58.090644 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:58.089933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.3:51801)
0929 21:03:58.090394 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:58.090468 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":243,"threads_started":1}
I20260929 21:03:58.194363 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:58.193652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.1:49917)
0929 21:03:58.194091 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:58.194204 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":221,"threads_started":1}
I20260929 21:03:58.634732 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:58.634019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.2:52193)
0929 21:03:58.634474 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:58.634570 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":236,"threads_started":1}
I20260929 21:03:59.092168 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:59.091487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.3:46393)
0929 21:03:59.091904 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:59.091989 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":220,"threads_started":1}
I20260929 21:03:59.196018 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:59.195233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.1:37475)
0929 21:03:59.195730 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:59.195807 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":236,"threads_started":1}
I20260929 21:03:59.636525 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:03:59.635750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.2:41351)
0929 21:03:59.636231 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0929 21:03:59.636307 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":228,"threads_started":1}
I20260929 21:04:00.093926 17442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:00.093094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.3:35759)
0929 21:04:00.093623 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:00.093705 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":239,"threads_started":1}
I20260929 21:04:00.197685 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:00.196993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.1:37635)
0929 21:04:00.197446 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:00.197524 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":235,"threads_started":1}
I20260929 21:04:00.638067 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:00.637384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.2:37779)
0929 21:04:00.637826 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:00.637902 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":248,"threads_started":1}
I20260929 21:04:01.095638 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:01.094930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.3:55903)
0929 21:04:01.095370 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:01.095450 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":226,"threads_started":1}
I20260929 21:04:01.199301 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:01.198588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.1:42431)
0929 21:04:01.199046 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:01.199129 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":219,"threads_started":1}
I20260929 21:04:01.639611 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:01.638926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.2:55735)
0929 21:04:01.639362 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:01.639443 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":214,"threads_started":1}
I20260929 21:04:02.097312 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:02.096521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.3:42011)
0929 21:04:02.097076 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:02.097152 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":285,"threads_started":1}
I20260929 21:04:02.200991 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:02.200227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.1:51725)
0929 21:04:02.200641 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:02.200717 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":224,"threads_started":1}
I20260929 21:04:02.641297 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:02.640537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.2:40689)
0929 21:04:02.641054 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:02.641135 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":255,"threads_started":1}
I20260929 21:04:03.098798 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:03.098136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.3:59277)
0929 21:04:03.098571 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:03.098647 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":231,"threads_started":1}
I20260929 21:04:03.202653 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:03.201901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.1:59015)
0929 21:04:03.202345 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:03.202457 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":226,"threads_started":1}
I20260929 21:04:03.642943 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:03.642251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.2:59031)
0929 21:04:03.642671 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:03.642754 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":207,"threads_started":1}
I20260929 21:04:04.100445 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:04.099706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.3:41621)
0929 21:04:04.100182 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:04.100270 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":207,"threads_started":1}
I20260929 21:04:04.204547 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:04.203707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.1:37241)
0929 21:04:04.204232 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:04.204359 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":225,"threads_started":1}
I20260929 21:04:04.644395 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:04.643762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.2:34135)
0929 21:04:04.644171 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:04.644245 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":205,"threads_started":1}
I20260929 21:04:05.102064 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:05.101322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.3:51605)
0929 21:04:05.101835 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:05.101911 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":298,"threads_started":1}
I20260929 21:04:05.206298 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:05.205448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.1:58837)
0929 21:04:05.205950 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:05.206038 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":235,"threads_started":1}
I20260929 21:04:05.646085 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:05.645276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.2:40395)
0929 21:04:05.645814 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:05.645895 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":306,"threads_started":1}
I20260929 21:04:06.103595 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:06.102866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.3:47595)
0929 21:04:06.103308 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:06.103388 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":229,"threads_started":1}
I20260929 21:04:06.208055 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:06.207331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.1:48321)
0929 21:04:06.207791 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:06.207869 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":246,"threads_started":1}
I20260929 21:04:06.647706 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:06.646994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.2:60575)
0929 21:04:06.647434 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:06.647515 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":225,"threads_started":1}
I20260929 21:04:07.105129 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:07.104403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.3:40725)
0929 21:04:07.104804 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:07.104941 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":207,"threads_started":1}
I20260929 21:04:07.209820 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:07.209145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.1:47473)
0929 21:04:07.209568 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:07.209643 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":217,"threads_started":1}
I20260929 21:04:07.649606 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:07.648886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.2:53911)
0929 21:04:07.649335 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:07.649408 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":209,"threads_started":1}
I20260929 21:04:08.091696 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 16988
I20260929 21:04:08.106886 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:08.106191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.3:56513)
0929 21:04:08.106623 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:08.106711 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":210,"threads_started":1}
I20260929 21:04:08.211663 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:08.210797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.1:46191)
0929 21:04:08.211344 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:08.211462 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":263,"threads_started":1}
I20260929 21:04:08.651494 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:08.650720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46807 (local address 127.0.97.2:36177)
0929 21:04:08.651242 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:08.651335 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":287,"threads_started":1}
I20260929 21:04:08.781883 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:08.092952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59656 (local address 127.0.97.1:46765)
0929 21:04:08.093503 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:08.093517 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:08.765537 (+672020us) server_negotiation.cc:407] Connection header received
0929 21:04:08.767218 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:08.767229 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:08.767672 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:08.768036 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:08.769649 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:08.770858 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:08.772161 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:08.772816 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:08.773087 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:08.775717 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:08.775724 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:08.775739 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:04:08.775785 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:08.778632 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:08.779290 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:08.779300 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:08.779313 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:08.779475 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:08.779868 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:08.779874 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:08.779879 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:08.780383 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:08.780686 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:08.781156 (+ 470us) server_negotiation.cc:299] Negotiation successful
0929 21:04:08.781580 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":274,"threads_started":1}
I20260929 21:04:08.783285 17151 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:04:08.866590 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 16988
I20260929 21:04:08.883296 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 17159
I20260929 21:04:08.900269 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:08.884263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53190 (local address 127.0.97.2:33721)
0929 21:04:08.884787 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:08.884801 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:08.884898 (+ 97us) server_negotiation.cc:407] Connection header received
0929 21:04:08.886518 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:08.886526 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:08.886952 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:08.887232 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:08.888508 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:08.889931 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:08.891162 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:08.891836 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:08.891966 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:08.894608 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:08.894615 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:08.894630 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:04:08.894663 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:08.897446 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:08.898096 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:08.898102 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:08.898109 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:08.898255 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:08.898562 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:08.898566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:08.898580 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:08.899049 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:08.899257 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:08.899625 (+ 368us) server_negotiation.cc:299] Negotiation successful
0929 21:04:08.899932 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":212,"threads_started":1}
I20260929 21:04:08.901773 17282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:04:08.984269 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 17159
I20260929 21:04:09.001847 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 17290
I20260929 21:04:09.018982 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:09.002781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50294 (local address 127.0.97.3:36811)
0929 21:04:09.003359 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:09.003379 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:09.003426 (+ 47us) server_negotiation.cc:407] Connection header received
0929 21:04:09.005056 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:09.005067 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:09.005523 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:09.005819 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:09.007073 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:09.008330 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:09.009698 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:09.010395 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:09.010552 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:09.013086 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:09.013095 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:09.013112 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:04:09.013149 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:09.016028 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:09.016666 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:09.016672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:09.016678 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:09.016922 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:09.017280 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:09.017283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:09.017285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:09.017791 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:09.018006 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:09.018370 (+ 364us) server_negotiation.cc:299] Negotiation successful
0929 21:04:09.018703 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":295,"threads_started":1}
I20260929 21:04:09.020288 17413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:04:09.102874 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 17290
2026-09-29T21:04:09Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64882 ms)
[----------] 30 tests from SecurityITest (292493 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 21:04:09 dist-test-slave-1dms krb5kdc[17477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 21:04:09 dist-test-slave-1dms krb5kdc[17477](info): set up 2 sockets
Sep 29 21:04:09 dist-test-slave-1dms krb5kdc[17477](info): commencing operation
krb5kdc: starting...
W20260929 21:04:10.200798 388 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 21:04:10 dist-test-slave-1dms krb5kdc[17477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T21:04:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:04:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 21:04:10.437016 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41401
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41401
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:04:10.766057 17490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:10.766355 17490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:10.766518 17490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:10.773522 17490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:04:10.773615 17490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:10.773716 17490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:10.773814 17490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:04:10.773877 17490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:04:10.783859 17490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:10.785235 17490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:10.786531 17490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:10.798664 17495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:10.798743 17496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:10.798699 17498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:10.799216 17490 server_base.cc:1034] running on GCE node
I20260929 21:04:10.799995 17490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:10.801415 17490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:10.802619 17490 hybrid_clock.cc:648] HybridClock initialized: now 1790715850802566 us; error 41 us; skew 500 ppm
I20260929 21:04:10.802969 17490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:04:10 dist-test-slave-1dms krb5kdc[17477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715850, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:04:10.805972 17490 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 21:04:10.807468 17490 webserver.cc:460] Webserver started at http://127.0.97.62:42793/ using document root <none> and password file <none>
I20260929 21:04:10.808133 17490 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:10.808269 17490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:10.808566 17490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:10.811483 17490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "e336c942fab34021b1f401b0e48142b9"
format_stamp: "Formatted at 2026-09-29 21:04:10 on dist-test-slave-1dms"
server_key: "2f406764588152e734d10b74d3ffc64e"
server_key_iv: "75152d674815cd03dc2ff6873c8eaaae"
server_key_version: "encryptionkey@0"
I20260929 21:04:10.812204 17490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "e336c942fab34021b1f401b0e48142b9"
format_stamp: "Formatted at 2026-09-29 21:04:10 on dist-test-slave-1dms"
server_key: "2f406764588152e734d10b74d3ffc64e"
server_key_iv: "75152d674815cd03dc2ff6873c8eaaae"
server_key_version: "encryptionkey@0"
I20260929 21:04:10.817131 17490 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 21:04:10.820312 17505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:10.821653 17490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:04:10.821832 17490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "e336c942fab34021b1f401b0e48142b9"
format_stamp: "Formatted at 2026-09-29 21:04:10 on dist-test-slave-1dms"
server_key: "2f406764588152e734d10b74d3ffc64e"
server_key_iv: "75152d674815cd03dc2ff6873c8eaaae"
server_key_version: "encryptionkey@0"
I20260929 21:04:10.822036 17490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:10.839588 17490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:10.843619 17490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:10.843895 17490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:10.861927 17490 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41401
I20260929 21:04:10.861958 17556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41401 every 8 connection(s)
I20260929 21:04:10.863224 17490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:04:10.866881 17557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:04:10.869620 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 17490
I20260929 21:04:10.869838 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:04:10.870635 388 external_mini_cluster.cc:1428] Setting key 056a4d4e72ab78cd1efb215ef9d5ec64
I20260929 21:04:10.878446 17557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e336c942fab34021b1f401b0e48142b9: Bootstrap starting.
I20260929 21:04:10.881397 17557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e336c942fab34021b1f401b0e48142b9: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:10.882417 17557 log.cc:824] T 00000000000000000000000000000000 P e336c942fab34021b1f401b0e48142b9: Log is configured to *not* fsync() on all Append() calls
Sep 29 21:04:10 dist-test-slave-1dms krb5kdc[17477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:04:10.885409 17557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e336c942fab34021b1f401b0e48142b9: No bootstrap required, opened a new log
I20260929 21:04:10.892521 17560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:10.872731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33852 (local address 127.0.97.62:41401)
0929 21:04:10.873457 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:10.873468 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:10.873507 (+ 39us) server_negotiation.cc:407] Connection header received
0929 21:04:10.875159 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:10.875180 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:10.875548 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:10.875936 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:10.877459 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:10.878807 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:10.880128 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:10.880754 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:10.880955 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:10.885580 (+ 4625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:10.885590 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:10.885611 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:04:10.885650 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:10.888649 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:10.889352 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:10.889360 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:10.889376 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:10.889538 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:10.889913 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:10.889923 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:10.889925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:10.890412 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:10.890735 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:10.891175 (+ 440us) server_negotiation.cc:299] Negotiation successful
0929 21:04:10.891577 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":301,"threads_started":1}
I20260929 21:04:10.893560 17557 raft_consensus.cc:348] T 00000000000000000000000000000000 P e336c942fab34021b1f401b0e48142b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e336c942fab34021b1f401b0e48142b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41401 } }
I20260929 21:04:10.894032 17557 raft_consensus.cc:374] T 00000000000000000000000000000000 P e336c942fab34021b1f401b0e48142b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:10.894141 17557 raft_consensus.cc:729] T 00000000000000000000000000000000 P e336c942fab34021b1f401b0e48142b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e336c942fab34021b1f401b0e48142b9, State: Initialized, Role: FOLLOWER
I20260929 21:04:10.894747 17557 consensus_queue.cc:260] T 00000000000000000000000000000000 P e336c942fab34021b1f401b0e48142b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e336c942fab34021b1f401b0e48142b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41401 } }
I20260929 21:04:10.894997 17557 raft_consensus.cc:388] T 00000000000000000000000000000000 P e336c942fab34021b1f401b0e48142b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:04:10.895152 17557 raft_consensus.cc:482] T 00000000000000000000000000000000 P e336c942fab34021b1f401b0e48142b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:04:10.895336 17557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e336c942fab34021b1f401b0e48142b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:10.897567 17557 raft_consensus.cc:504] T 00000000000000000000000000000000 P e336c942fab34021b1f401b0e48142b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e336c942fab34021b1f401b0e48142b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41401 } }
I20260929 21:04:10.898036 17557 leader_election.cc:302] T 00000000000000000000000000000000 P e336c942fab34021b1f401b0e48142b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e336c942fab34021b1f401b0e48142b9; no voters:
I20260929 21:04:10.898517 17557 leader_election.cc:288] T 00000000000000000000000000000000 P e336c942fab34021b1f401b0e48142b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 21:04:10.898645 17562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e336c942fab34021b1f401b0e48142b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:04:10.899263 17562 raft_consensus.cc:686] T 00000000000000000000000000000000 P e336c942fab34021b1f401b0e48142b9 [term 1 LEADER]: Becoming Leader. State: Replica: e336c942fab34021b1f401b0e48142b9, State: Running, Role: LEADER
I20260929 21:04:10.899860 17562 consensus_queue.cc:237] T 00000000000000000000000000000000 P e336c942fab34021b1f401b0e48142b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e336c942fab34021b1f401b0e48142b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41401 } }
I20260929 21:04:10.901394 17557 sys_catalog.cc:564] T 00000000000000000000000000000000 P e336c942fab34021b1f401b0e48142b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:04:10.902843 17564 sys_catalog.cc:455] T 00000000000000000000000000000000 P e336c942fab34021b1f401b0e48142b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e336c942fab34021b1f401b0e48142b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e336c942fab34021b1f401b0e48142b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41401 } } }
I20260929 21:04:10.903194 17564 sys_catalog.cc:458] T 00000000000000000000000000000000 P e336c942fab34021b1f401b0e48142b9 [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:10.902622 17563 sys_catalog.cc:455] T 00000000000000000000000000000000 P e336c942fab34021b1f401b0e48142b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e336c942fab34021b1f401b0e48142b9. Latest consensus state: current_term: 1 leader_uuid: "e336c942fab34021b1f401b0e48142b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e336c942fab34021b1f401b0e48142b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41401 } } }
I20260929 21:04:10.903774 17563 sys_catalog.cc:458] T 00000000000000000000000000000000 P e336c942fab34021b1f401b0e48142b9 [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:10.904873 17567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:04:10.910871 17567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:04:10.919404 17567 catalog_manager.cc:1347] Generated new cluster ID: 7c5470112d4b49a4a217554b38bf6f5a
I20260929 21:04:10.919605 17567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:04:10.942837 17567 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:04:10.944697 17567 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:04:10.953058 17567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e336c942fab34021b1f401b0e48142b9: Generated new TSK 0
I20260929 21:04:10.954049 17567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 21:04:11.016043 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41401
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:04:11.341122 17583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:11.341379 17583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:11.341580 17583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:11.348685 17583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:11.348807 17583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:11.348999 17583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 21:04:11.358583 17583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:11.359781 17583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:11.361073 17583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:11.373579 17589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:11.373652 17588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:11.373652 17591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:11.374519 17583 server_base.cc:1034] running on GCE node
I20260929 21:04:11.375056 17583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:11.375847 17583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:11.377048 17583 hybrid_clock.cc:648] HybridClock initialized: now 1790715851376999 us; error 33 us; skew 500 ppm
I20260929 21:04:11.377358 17583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:04:11 dist-test-slave-1dms krb5kdc[17477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715851, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:04:11.380388 17583 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 21:04:11.381896 17583 webserver.cc:460] Webserver started at http://127.0.97.1:41899/ using document root <none> and password file <none>
I20260929 21:04:11.382575 17583 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:11.382697 17583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:11.383008 17583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:11.385880 17583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-0/data/instance:
uuid: "3ec455833a2d41508940ed31beb2abce"
format_stamp: "Formatted at 2026-09-29 21:04:11 on dist-test-slave-1dms"
server_key: "5e6f72935390062dbaa8d268dee639e7"
server_key_iv: "2c09dff9d9ca83339e4b5a4d4456dd02"
server_key_version: "encryptionkey@0"
I20260929 21:04:11.386648 17583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3ec455833a2d41508940ed31beb2abce"
format_stamp: "Formatted at 2026-09-29 21:04:11 on dist-test-slave-1dms"
server_key: "5e6f72935390062dbaa8d268dee639e7"
server_key_iv: "2c09dff9d9ca83339e4b5a4d4456dd02"
server_key_version: "encryptionkey@0"
I20260929 21:04:11.391511 17583 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 21:04:11.394752 17598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:11.396000 17583 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 21:04:11.396167 17583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-0/wal
uuid: "3ec455833a2d41508940ed31beb2abce"
format_stamp: "Formatted at 2026-09-29 21:04:11 on dist-test-slave-1dms"
server_key: "5e6f72935390062dbaa8d268dee639e7"
server_key_iv: "2c09dff9d9ca83339e4b5a4d4456dd02"
server_key_version: "encryptionkey@0"
I20260929 21:04:11.396363 17583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:11.420058 17583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:11.424031 17583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:11.424332 17583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:11.425387 17583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:04:11.426793 17583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:04:11.426890 17583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:11.427035 17583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:04:11.427104 17583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:11.459237 17583 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42831
I20260929 21:04:11.459295 17710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42831 every 8 connection(s)
I20260929 21:04:11.460512 17583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
I20260929 21:04:11.470896 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 17583
I20260929 21:04:11.471159 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-0/wal/instance
I20260929 21:04:11.471879 388 external_mini_cluster.cc:1428] Setting key 744558b979ba2c079082f842f4cc13cd
Sep 29 21:04:11 dist-test-slave-1dms krb5kdc[17477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715851, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:04:11.479737 17713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:11.462906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60113 (local address 127.0.97.62:41401)
0929 21:04:11.463337 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:11.463341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:11.464167 (+ 826us) server_negotiation.cc:407] Connection header received
0929 21:04:11.466012 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:11.466019 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:11.466117 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:11.466229 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:11.467672 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:11.468413 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:11.469968 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:11.470366 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:11.470531 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:11.474298 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:11.474303 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:11.474306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:04:11.474335 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:11.476817 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:11.477543 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:11.477549 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:11.477560 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:11.477674 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:11.478075 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:11.478080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:11.478084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:11.478331 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:11.478481 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:11.479145 (+ 664us) server_negotiation.cc:299] Negotiation successful
0929 21:04:11.479371 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":247,"threads_started":1}
I20260929 21:04:11.480677 17714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:11.463348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41401 (local address 127.0.97.1:60113)
0929 21:04:11.464014 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:11.464050 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:11.465757 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:11.466477 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:11.466485 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:11.466971 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:04:11.467502 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:11.467519 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:11.468550 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:11.468564 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:11.469827 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:11.469839 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:11.470595 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:11.470603 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:11.470840 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:11.471572 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:04:11.471593 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:04:11.474100 (+ 2507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:04:11.476987 (+ 2887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:04:11.476995 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:04:11.477011 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:04:11.477422 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:04:11.477778 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:04:11.477783 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:04:11.477787 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:04:11.477962 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:04:11.478451 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:04:11.478459 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:04:11.478783 (+ 324us) client_negotiation.cc:769] Sending connection context
0929 21:04:11.479031 (+ 248us) client_negotiation.cc:240] Negotiation successful
0929 21:04:11.479549 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":257,"threads_started":1}
I20260929 21:04:11.482275 17711 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41401
I20260929 21:04:11.482563 17711 heartbeater.cc:464] Registering TS with master...
I20260929 21:04:11.483237 17711 heartbeater.cc:511] Master 127.0.97.62:41401 requested a full tablet report, sending...
I20260929 21:04:11.485225 17522 ts_manager.cc:194] Registered new tserver with Master: 3ec455833a2d41508940ed31beb2abce (127.0.97.1:42831)
I20260929 21:04:11.486644 17522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:60113
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 21:04:11.524181 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41401
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:04:11.843291 17717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:11.843577 17717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:11.843670 17717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:11.850858 17717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:11.850956 17717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:11.851070 17717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 21:04:11.860478 17717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:11.861763 17717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:11.863047 17717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:11.876058 17723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:11.876094 17725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:11.876924 17722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:11.877983 17717 server_base.cc:1034] running on GCE node
I20260929 21:04:11.878643 17717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:11.879637 17717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:11.880908 17717 hybrid_clock.cc:648] HybridClock initialized: now 1790715851880851 us; error 45 us; skew 500 ppm
I20260929 21:04:11.881228 17717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:04:11 dist-test-slave-1dms krb5kdc[17477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715851, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:04:11.884274 17717 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 21:04:11.885722 17717 webserver.cc:460] Webserver started at http://127.0.97.2:39227/ using document root <none> and password file <none>
I20260929 21:04:11.886361 17717 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:11.886472 17717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:11.886718 17717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:11.889554 17717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-1/data/instance:
uuid: "0cd69134625943ec86a2a7440139c0a0"
format_stamp: "Formatted at 2026-09-29 21:04:11 on dist-test-slave-1dms"
server_key: "3136361020c32e53434c208f9650f044"
server_key_iv: "a01ea8020ed27c9f71f17754688acac2"
server_key_version: "encryptionkey@0"
I20260929 21:04:11.890269 17717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0cd69134625943ec86a2a7440139c0a0"
format_stamp: "Formatted at 2026-09-29 21:04:11 on dist-test-slave-1dms"
server_key: "3136361020c32e53434c208f9650f044"
server_key_iv: "a01ea8020ed27c9f71f17754688acac2"
server_key_version: "encryptionkey@0"
I20260929 21:04:11.895243 17717 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 21:04:11.898649 17732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:11.900082 17717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 21:04:11.900213 17717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-1/wal
uuid: "0cd69134625943ec86a2a7440139c0a0"
format_stamp: "Formatted at 2026-09-29 21:04:11 on dist-test-slave-1dms"
server_key: "3136361020c32e53434c208f9650f044"
server_key_iv: "a01ea8020ed27c9f71f17754688acac2"
server_key_version: "encryptionkey@0"
I20260929 21:04:11.900415 17717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:11.915848 17717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:11.919863 17717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:11.920149 17717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:11.921176 17717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:04:11.922497 17717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:04:11.922602 17717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:11.922736 17717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:04:11.922804 17717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:11.956420 17717 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33075
I20260929 21:04:11.956521 17844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33075 every 8 connection(s)
I20260929 21:04:11.957739 17717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
I20260929 21:04:11.966858 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 17717
I20260929 21:04:11.967038 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-1/wal/instance
I20260929 21:04:11.967571 388 external_mini_cluster.cc:1428] Setting key 1b1c1c3a0ae9047969660aa5bc7ada6e
Sep 29 21:04:11 dist-test-slave-1dms krb5kdc[17477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715851, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:04:11.977954 17713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:11.960310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46409 (local address 127.0.97.62:41401)
0929 21:04:11.960521 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:11.960528 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:11.961738 (+ 1210us) server_negotiation.cc:407] Connection header received
0929 21:04:11.963613 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:11.963619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:11.963694 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:11.963787 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:11.965321 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:11.966037 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:11.968293 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:11.968953 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:11.969092 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:11.972421 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:11.972428 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:11.972432 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:04:11.972472 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:11.974991 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:11.975706 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:11.975713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:11.975717 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:11.975845 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:11.976250 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:11.976256 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:11.976260 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:11.976531 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:11.976681 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:11.977389 (+ 708us) server_negotiation.cc:299] Negotiation successful
0929 21:04:11.977611 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 21:04:11.979056 17847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:11.960827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41401 (local address 127.0.97.2:46409)
0929 21:04:11.961577 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:11.961619 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:11.963347 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:11.964015 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:11.964023 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:11.964529 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:04:11.965156 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:11.965173 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:11.966178 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:11.966182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:11.968152 (+ 1970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:11.968165 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:11.969176 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:11.969183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:11.969388 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:11.970090 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:04:11.970117 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:04:11.972212 (+ 2095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:04:11.975138 (+ 2926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:04:11.975148 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:04:11.975164 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:04:11.975593 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:04:11.975958 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:04:11.975963 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:04:11.975966 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:04:11.976147 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:04:11.976655 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:04:11.976663 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:04:11.977047 (+ 384us) client_negotiation.cc:769] Sending connection context
0929 21:04:11.977306 (+ 259us) client_negotiation.cc:240] Negotiation successful
0929 21:04:11.977836 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":264,"threads_started":1}
I20260929 21:04:11.980531 17845 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41401
I20260929 21:04:11.980876 17845 heartbeater.cc:464] Registering TS with master...
I20260929 21:04:11.981580 17845 heartbeater.cc:511] Master 127.0.97.62:41401 requested a full tablet report, sending...
I20260929 21:04:11.983019 17522 ts_manager.cc:194] Registered new tserver with Master: 0cd69134625943ec86a2a7440139c0a0 (127.0.97.2:33075)
I20260929 21:04:11.984274 17522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:46409
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 21:04:12.020452 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41401
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:04:12.343153 17850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:12.343413 17850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:12.343513 17850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:12.350620 17850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:12.350739 17850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:12.350858 17850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 21:04:12.360375 17850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:12.361629 17850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:12.362833 17850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:12.375427 17856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:12.375473 17855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:12.375504 17858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:12.376593 17850 server_base.cc:1034] running on GCE node
I20260929 21:04:12.377204 17850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:12.378072 17850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:12.379287 17850 hybrid_clock.cc:648] HybridClock initialized: now 1790715852379229 us; error 56 us; skew 500 ppm
I20260929 21:04:12.379587 17850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:04:12 dist-test-slave-1dms krb5kdc[17477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:04:12.382627 17850 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 21:04:12.383994 17850 webserver.cc:460] Webserver started at http://127.0.97.3:37361/ using document root <none> and password file <none>
I20260929 21:04:12.384617 17850 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:12.384711 17850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:12.385036 17850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:12.387774 17850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-2/data/instance:
uuid: "d614d61d0ef94dd897d302f86a3efe3e"
format_stamp: "Formatted at 2026-09-29 21:04:12 on dist-test-slave-1dms"
server_key: "a965ce0f36472a029144c020032c2948"
server_key_iv: "eee854e6e1b7e1aa2cf9c99af17ad64f"
server_key_version: "encryptionkey@0"
I20260929 21:04:12.388471 17850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d614d61d0ef94dd897d302f86a3efe3e"
format_stamp: "Formatted at 2026-09-29 21:04:12 on dist-test-slave-1dms"
server_key: "a965ce0f36472a029144c020032c2948"
server_key_iv: "eee854e6e1b7e1aa2cf9c99af17ad64f"
server_key_version: "encryptionkey@0"
I20260929 21:04:12.393318 17850 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 21:04:12.396610 17865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:12.397920 17850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:04:12.398072 17850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-2/wal
uuid: "d614d61d0ef94dd897d302f86a3efe3e"
format_stamp: "Formatted at 2026-09-29 21:04:12 on dist-test-slave-1dms"
server_key: "a965ce0f36472a029144c020032c2948"
server_key_iv: "eee854e6e1b7e1aa2cf9c99af17ad64f"
server_key_version: "encryptionkey@0"
I20260929 21:04:12.398272 17850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:12.422762 17850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:12.427253 17850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:12.427505 17850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:12.428372 17850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:04:12.429713 17850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:04:12.429783 17850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:12.429867 17850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:04:12.429912 17850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:12.465049 17850 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38605
I20260929 21:04:12.465133 17977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38605 every 8 connection(s)
I20260929 21:04:12.466321 17850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
I20260929 21:04:12.472805 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 17850
I20260929 21:04:12.473011 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-2/wal/instance
I20260929 21:04:12.473436 388 external_mini_cluster.cc:1428] Setting key 834fe4251c6d0028bb6eea0a29060362
Sep 29 21:04:12 dist-test-slave-1dms krb5kdc[17477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:04:12.486760 17713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:12.468742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48885 (local address 127.0.97.62:41401)
0929 21:04:12.468978 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:12.468984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:12.470190 (+ 1206us) server_negotiation.cc:407] Connection header received
0929 21:04:12.472180 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:12.472185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:12.472252 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:12.472339 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:12.474135 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:12.475011 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:12.477016 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:12.477401 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:12.477496 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:12.481000 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:12.481005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:12.481009 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:04:12.481044 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:12.483864 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:12.484555 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:12.484559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:12.484562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:12.484697 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:12.485240 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:12.485244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:12.485246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:12.485518 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:12.485657 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:12.486260 (+ 603us) server_negotiation.cc:299] Negotiation successful
0929 21:04:12.486440 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 21:04:12.487823 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:12.469288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41401 (local address 127.0.97.3:48885)
0929 21:04:12.470032 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:12.470075 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:12.471922 (+ 1847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:12.472597 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:12.472605 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:12.473301 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:04:12.473947 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:12.473967 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:12.475167 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:12.475171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:12.476869 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:12.476889 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:12.477582 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:12.477586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:12.477793 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:12.478502 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:04:12.478525 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:04:12.480741 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:04:12.483992 (+ 3251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:04:12.483998 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:04:12.484010 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:04:12.484415 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:04:12.484801 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:04:12.484804 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:04:12.484806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:04:12.485027 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:04:12.485633 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:04:12.485639 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:04:12.485923 (+ 284us) client_negotiation.cc:769] Sending connection context
0929 21:04:12.486152 (+ 229us) client_negotiation.cc:240] Negotiation successful
0929 21:04:12.486590 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":303,"threads_started":1}
I20260929 21:04:12.489147 17978 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41401
I20260929 21:04:12.489399 17978 heartbeater.cc:464] Registering TS with master...
I20260929 21:04:12.489419 17711 heartbeater.cc:503] Master 127.0.97.62:41401 was elected leader, sending a full tablet report...
I20260929 21:04:12.490191 17978 heartbeater.cc:511] Master 127.0.97.62:41401 requested a full tablet report, sending...
I20260929 21:04:12.491571 17522 ts_manager.cc:194] Registered new tserver with Master: d614d61d0ef94dd897d302f86a3efe3e (127.0.97.3:38605)
I20260929 21:04:12.492311 17522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48885
I20260929 21:04:12.500814 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 21:04:12.519908 17713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:12.507459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33858 (local address 127.0.97.62:41401)
0929 21:04:12.507706 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:12.507715 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:12.507787 (+ 72us) server_negotiation.cc:407] Connection header received
0929 21:04:12.507957 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:12.507963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:12.508182 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:12.508307 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:12.509794 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:12.510493 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:12.511773 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:12.512193 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:12.512322 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:12.514675 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:12.514681 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:12.514685 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:04:12.514721 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:12.517292 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:12.517986 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:12.517996 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:12.517998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:12.518112 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:12.518457 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:12.518464 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:12.518467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:12.518782 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:12.519048 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:12.519375 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 21:04:12.519618 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 21:04:12.523597 17522 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33858:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 21:04:12.530566 17522 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 21:04:12.555090 17991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:12.542147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38605 (local address 127.0.0.1:55398)
0929 21:04:12.544945 (+ 2798us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:12.544973 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:12.545185 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:12.546171 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:12.546178 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:12.546204 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:04:12.546623 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:12.546643 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:12.549262 (+ 2619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:12.549269 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:12.551863 (+ 2594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:12.551879 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:12.553971 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:12.553982 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:12.554242 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:12.554300 (+ 58us) client_negotiation.cc:769] Sending connection context
0929 21:04:12.554463 (+ 163us) client_negotiation.cc:240] Negotiation successful
0929 21:04:12.554658 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2659,"mutex_wait_us":24,"thread_start_us":2536,"threads_started":1}
I20260929 21:04:12.555080 17993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:12.542786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55398 (local address 127.0.97.3:38605)
0929 21:04:12.543668 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:12.543681 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:12.545078 (+ 1397us) server_negotiation.cc:407] Connection header received
0929 21:04:12.545383 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:12.545393 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:12.545643 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:12.545794 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:04:12.546870 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:12.549077 (+ 2207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:12.552013 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:12.553454 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:12.553582 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:12.553832 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:12.554519 (+ 687us) server_negotiation.cc:299] Negotiation successful
0929 21:04:12.554713 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":451,"threads_started":1}
I20260929 21:04:12.558576 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:12.545255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33075 (local address 127.0.0.1:43828)
0929 21:04:12.551238 (+ 5983us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:12.551260 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:12.551379 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:12.552053 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:12.552057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:12.552085 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:04:12.552360 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:12.552368 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:12.553965 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:12.553970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:12.555975 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:12.555986 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:12.557780 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:12.557786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:12.557939 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:12.557971 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 21:04:12.558078 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 21:04:12.558196 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5850,"mutex_wait_us":4,"thread_start_us":5711,"threads_started":1}
I20260929 21:04:12.559278 17995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:12.545706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43828 (local address 127.0.97.2:33075)
0929 21:04:12.550909 (+ 5203us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:12.550917 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:12.551518 (+ 601us) server_negotiation.cc:407] Connection header received
0929 21:04:12.551620 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:12.551627 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:12.551835 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:12.552014 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:04:12.552502 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:12.553784 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:12.556115 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:12.557566 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:12.557691 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:12.557877 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:12.558789 (+ 912us) server_negotiation.cc:299] Negotiation successful
0929 21:04:12.558946 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5053,"thread_start_us":4951,"threads_started":1}
I20260929 21:04:12.558987 17990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:12.541930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42831 (local address 127.0.0.1:52600)
0929 21:04:12.544210 (+ 2280us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:12.544244 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:12.544400 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:12.551269 (+ 6869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:12.551274 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:12.551298 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:04:12.551710 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:12.551719 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:12.553599 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:12.553604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:12.556110 (+ 2506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:12.556128 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:12.558142 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:12.558152 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:12.558358 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:12.558389 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 21:04:12.558491 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 21:04:12.558614 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2063,"mutex_wait_us":31,"thread_start_us":1908,"threads_started":1}
I20260929 21:04:12.559080 17913 tablet_service.cc:1467] Processing CreateTablet for tablet 3daef39b8eb240cfa6f032b182dc277a (DEFAULT_TABLE table=test-table [id=ed36b4f837f544c488dfc258279c7ee9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:04:12.560323 17992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:12.542240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52600 (local address 127.0.97.1:42831)
0929 21:04:12.550646 (+ 8406us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:12.550657 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:12.550687 (+ 30us) server_negotiation.cc:407] Connection header received
0929 21:04:12.550789 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:12.550797 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:12.551034 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:12.551302 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:04:12.551873 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:12.553427 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:12.556278 (+ 2851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:12.557688 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:12.557798 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:12.557999 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:12.559095 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0929 21:04:12.559231 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8225,"thread_start_us":8089,"threads_started":1}
I20260929 21:04:12.560491 17913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3daef39b8eb240cfa6f032b182dc277a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:04:12.563016 17646 tablet_service.cc:1467] Processing CreateTablet for tablet 3daef39b8eb240cfa6f032b182dc277a (DEFAULT_TABLE table=test-table [id=ed36b4f837f544c488dfc258279c7ee9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:04:12.562914 17780 tablet_service.cc:1467] Processing CreateTablet for tablet 3daef39b8eb240cfa6f032b182dc277a (DEFAULT_TABLE table=test-table [id=ed36b4f837f544c488dfc258279c7ee9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:04:12.564270 17780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3daef39b8eb240cfa6f032b182dc277a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:04:12.564319 17646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3daef39b8eb240cfa6f032b182dc277a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:04:12.575562 17996 tablet_bootstrap.cc:492] T 3daef39b8eb240cfa6f032b182dc277a P d614d61d0ef94dd897d302f86a3efe3e: Bootstrap starting.
I20260929 21:04:12.577334 17997 tablet_bootstrap.cc:492] T 3daef39b8eb240cfa6f032b182dc277a P 3ec455833a2d41508940ed31beb2abce: Bootstrap starting.
I20260929 21:04:12.579198 17996 tablet_bootstrap.cc:654] T 3daef39b8eb240cfa6f032b182dc277a P d614d61d0ef94dd897d302f86a3efe3e: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:12.580457 17996 log.cc:824] T 3daef39b8eb240cfa6f032b182dc277a P d614d61d0ef94dd897d302f86a3efe3e: Log is configured to *not* fsync() on all Append() calls
I20260929 21:04:12.581763 17997 tablet_bootstrap.cc:654] T 3daef39b8eb240cfa6f032b182dc277a P 3ec455833a2d41508940ed31beb2abce: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:12.582974 17997 log.cc:824] T 3daef39b8eb240cfa6f032b182dc277a P 3ec455833a2d41508940ed31beb2abce: Log is configured to *not* fsync() on all Append() calls
I20260929 21:04:12.583556 17996 tablet_bootstrap.cc:492] T 3daef39b8eb240cfa6f032b182dc277a P d614d61d0ef94dd897d302f86a3efe3e: No bootstrap required, opened a new log
I20260929 21:04:12.583819 17996 ts_tablet_manager.cc:1397] T 3daef39b8eb240cfa6f032b182dc277a P d614d61d0ef94dd897d302f86a3efe3e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 21:04:12.585489 17997 tablet_bootstrap.cc:492] T 3daef39b8eb240cfa6f032b182dc277a P 3ec455833a2d41508940ed31beb2abce: No bootstrap required, opened a new log
I20260929 21:04:12.585620 17998 tablet_bootstrap.cc:492] T 3daef39b8eb240cfa6f032b182dc277a P 0cd69134625943ec86a2a7440139c0a0: Bootstrap starting.
I20260929 21:04:12.585763 17997 ts_tablet_manager.cc:1397] T 3daef39b8eb240cfa6f032b182dc277a P 3ec455833a2d41508940ed31beb2abce: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260929 21:04:12.589092 17998 tablet_bootstrap.cc:654] T 3daef39b8eb240cfa6f032b182dc277a P 0cd69134625943ec86a2a7440139c0a0: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:12.590291 17998 log.cc:824] T 3daef39b8eb240cfa6f032b182dc277a P 0cd69134625943ec86a2a7440139c0a0: Log is configured to *not* fsync() on all Append() calls
I20260929 21:04:12.593444 17997 raft_consensus.cc:348] T 3daef39b8eb240cfa6f032b182dc277a P 3ec455833a2d41508940ed31beb2abce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec455833a2d41508940ed31beb2abce" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42831 } } peers { permanent_uuid: "d614d61d0ef94dd897d302f86a3efe3e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38605 } } peers { permanent_uuid: "0cd69134625943ec86a2a7440139c0a0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33075 } }
I20260929 21:04:12.593815 17997 raft_consensus.cc:374] T 3daef39b8eb240cfa6f032b182dc277a P 3ec455833a2d41508940ed31beb2abce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:12.593933 17997 raft_consensus.cc:729] T 3daef39b8eb240cfa6f032b182dc277a P 3ec455833a2d41508940ed31beb2abce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ec455833a2d41508940ed31beb2abce, State: Initialized, Role: FOLLOWER
I20260929 21:04:12.593873 17996 raft_consensus.cc:348] T 3daef39b8eb240cfa6f032b182dc277a P d614d61d0ef94dd897d302f86a3efe3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec455833a2d41508940ed31beb2abce" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42831 } } peers { permanent_uuid: "d614d61d0ef94dd897d302f86a3efe3e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38605 } } peers { permanent_uuid: "0cd69134625943ec86a2a7440139c0a0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33075 } }
I20260929 21:04:12.594388 17996 raft_consensus.cc:374] T 3daef39b8eb240cfa6f032b182dc277a P d614d61d0ef94dd897d302f86a3efe3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:12.594519 17998 tablet_bootstrap.cc:492] T 3daef39b8eb240cfa6f032b182dc277a P 0cd69134625943ec86a2a7440139c0a0: No bootstrap required, opened a new log
I20260929 21:04:12.594513 17996 raft_consensus.cc:729] T 3daef39b8eb240cfa6f032b182dc277a P d614d61d0ef94dd897d302f86a3efe3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d614d61d0ef94dd897d302f86a3efe3e, State: Initialized, Role: FOLLOWER
I20260929 21:04:12.594539 17997 consensus_queue.cc:260] T 3daef39b8eb240cfa6f032b182dc277a P 3ec455833a2d41508940ed31beb2abce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec455833a2d41508940ed31beb2abce" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42831 } } peers { permanent_uuid: "d614d61d0ef94dd897d302f86a3efe3e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38605 } } peers { permanent_uuid: "0cd69134625943ec86a2a7440139c0a0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33075 } }
I20260929 21:04:12.594795 17998 ts_tablet_manager.cc:1397] T 3daef39b8eb240cfa6f032b182dc277a P 0cd69134625943ec86a2a7440139c0a0: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260929 21:04:12.595249 17996 consensus_queue.cc:260] T 3daef39b8eb240cfa6f032b182dc277a P d614d61d0ef94dd897d302f86a3efe3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec455833a2d41508940ed31beb2abce" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42831 } } peers { permanent_uuid: "d614d61d0ef94dd897d302f86a3efe3e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38605 } } peers { permanent_uuid: "0cd69134625943ec86a2a7440139c0a0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33075 } }
I20260929 21:04:12.596284 17997 ts_tablet_manager.cc:1428] T 3daef39b8eb240cfa6f032b182dc277a P 3ec455833a2d41508940ed31beb2abce: Time spent starting tablet: real 0.010s user 0.007s sys 0.003s
I20260929 21:04:12.605571 17978 heartbeater.cc:503] Master 127.0.97.62:41401 was elected leader, sending a full tablet report...
I20260929 21:04:12.606388 17996 ts_tablet_manager.cc:1428] T 3daef39b8eb240cfa6f032b182dc277a P d614d61d0ef94dd897d302f86a3efe3e: Time spent starting tablet: real 0.022s user 0.012s sys 0.011s
I20260929 21:04:12.609225 17998 raft_consensus.cc:348] T 3daef39b8eb240cfa6f032b182dc277a P 0cd69134625943ec86a2a7440139c0a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec455833a2d41508940ed31beb2abce" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42831 } } peers { permanent_uuid: "d614d61d0ef94dd897d302f86a3efe3e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38605 } } peers { permanent_uuid: "0cd69134625943ec86a2a7440139c0a0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33075 } }
I20260929 21:04:12.609613 17998 raft_consensus.cc:374] T 3daef39b8eb240cfa6f032b182dc277a P 0cd69134625943ec86a2a7440139c0a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:12.609737 17998 raft_consensus.cc:729] T 3daef39b8eb240cfa6f032b182dc277a P 0cd69134625943ec86a2a7440139c0a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cd69134625943ec86a2a7440139c0a0, State: Initialized, Role: FOLLOWER
I20260929 21:04:12.610281 17998 consensus_queue.cc:260] T 3daef39b8eb240cfa6f032b182dc277a P 0cd69134625943ec86a2a7440139c0a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec455833a2d41508940ed31beb2abce" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42831 } } peers { permanent_uuid: "d614d61d0ef94dd897d302f86a3efe3e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38605 } } peers { permanent_uuid: "0cd69134625943ec86a2a7440139c0a0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33075 } }
I20260929 21:04:12.611248 17845 heartbeater.cc:503] Master 127.0.97.62:41401 was elected leader, sending a full tablet report...
I20260929 21:04:12.611696 17998 ts_tablet_manager.cc:1428] T 3daef39b8eb240cfa6f032b182dc277a P 0cd69134625943ec86a2a7440139c0a0: Time spent starting tablet: real 0.017s user 0.017s sys 0.000s
W20260929 21:04:12.709785 17846 tablet.cc:2369] T 3daef39b8eb240cfa6f032b182dc277a P 0cd69134625943ec86a2a7440139c0a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 21:04:12.712873 17712 tablet.cc:2369] T 3daef39b8eb240cfa6f032b182dc277a P 3ec455833a2d41508940ed31beb2abce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 21:04:12.713307 18002 raft_consensus.cc:482] T 3daef39b8eb240cfa6f032b182dc277a P 3ec455833a2d41508940ed31beb2abce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 21:04:12.713611 18002 raft_consensus.cc:504] T 3daef39b8eb240cfa6f032b182dc277a P 3ec455833a2d41508940ed31beb2abce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec455833a2d41508940ed31beb2abce" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42831 } } peers { permanent_uuid: "d614d61d0ef94dd897d302f86a3efe3e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38605 } } peers { permanent_uuid: "0cd69134625943ec86a2a7440139c0a0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33075 } }
I20260929 21:04:12.715245 18002 leader_election.cc:288] T 3daef39b8eb240cfa6f032b182dc277a P 3ec455833a2d41508940ed31beb2abce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d614d61d0ef94dd897d302f86a3efe3e (127.0.97.3:38605), 0cd69134625943ec86a2a7440139c0a0 (127.0.97.2:33075)
W20260929 21:04:12.717957 17979 tablet.cc:2369] T 3daef39b8eb240cfa6f032b182dc277a P d614d61d0ef94dd897d302f86a3efe3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 21:04:12.723959 17995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:12.716493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50561 (local address 127.0.97.2:33075)
0929 21:04:12.716692 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:12.716698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:12.717041 (+ 343us) server_negotiation.cc:407] Connection header received
0929 21:04:12.717183 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:12.717191 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:12.717266 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:12.717363 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:04:12.717933 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:12.718962 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:12.721461 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:12.722720 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:12.722859 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:12.722954 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:12.723576 (+ 622us) server_negotiation.cc:299] Negotiation successful
0929 21:04:12.723706 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 21:04:12.723959 17993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:12.715664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42073 (local address 127.0.97.3:38605)
0929 21:04:12.715895 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:12.715906 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:12.716378 (+ 472us) server_negotiation.cc:407] Connection header received
0929 21:04:12.716667 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:12.716673 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:12.716764 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:12.716816 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:04:12.717634 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:12.718799 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:12.721461 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:12.722720 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:12.722859 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:12.722958 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:12.723589 (+ 631us) server_negotiation.cc:299] Negotiation successful
0929 21:04:12.723706 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 21:04:12.724185 18005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:12.715632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38605 (local address 127.0.97.1:42073)
0929 21:04:12.716225 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:12.716242 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:12.716438 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:12.717049 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:12.717054 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:12.717084 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:04:12.717491 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:12.717503 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:12.718917 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:12.718921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:12.721295 (+ 2374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:12.721308 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:12.722990 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:12.722998 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:12.723282 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:12.723440 (+ 158us) client_negotiation.cc:769] Sending connection context
0929 21:04:12.723537 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 21:04:12.723704 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":190,"thread_start_us":333,"threads_started":1}
I20260929 21:04:12.724210 18006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:12.716320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33075 (local address 127.0.97.1:50561)
0929 21:04:12.716947 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:12.716960 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:12.717068 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:12.717372 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:12.717375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:12.717395 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:04:12.717768 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:12.717780 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:12.719098 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:12.719104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:12.721293 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:12.721308 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:12.723006 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:12.723011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:12.723282 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:12.723440 (+ 158us) client_negotiation.cc:769] Sending connection context
0929 21:04:12.723537 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 21:04:12.723704 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":143,"thread_start_us":302,"threads_started":1}
I20260929 21:04:12.725380 17800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3daef39b8eb240cfa6f032b182dc277a" candidate_uuid: "3ec455833a2d41508940ed31beb2abce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cd69134625943ec86a2a7440139c0a0" is_pre_election: true
I20260929 21:04:12.725384 17933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3daef39b8eb240cfa6f032b182dc277a" candidate_uuid: "3ec455833a2d41508940ed31beb2abce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d614d61d0ef94dd897d302f86a3efe3e" is_pre_election: true
I20260929 21:04:12.725943 17800 raft_consensus.cc:2454] T 3daef39b8eb240cfa6f032b182dc277a P 0cd69134625943ec86a2a7440139c0a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ec455833a2d41508940ed31beb2abce in term 0.
I20260929 21:04:12.725943 17933 raft_consensus.cc:2454] T 3daef39b8eb240cfa6f032b182dc277a P d614d61d0ef94dd897d302f86a3efe3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ec455833a2d41508940ed31beb2abce in term 0.
I20260929 21:04:12.726603 17599 leader_election.cc:302] T 3daef39b8eb240cfa6f032b182dc277a P 3ec455833a2d41508940ed31beb2abce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ec455833a2d41508940ed31beb2abce, d614d61d0ef94dd897d302f86a3efe3e; no voters:
I20260929 21:04:12.726949 18002 raft_consensus.cc:2781] T 3daef39b8eb240cfa6f032b182dc277a P 3ec455833a2d41508940ed31beb2abce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 21:04:12.727098 18002 raft_consensus.cc:482] T 3daef39b8eb240cfa6f032b182dc277a P 3ec455833a2d41508940ed31beb2abce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 21:04:12.727209 18002 raft_consensus.cc:3037] T 3daef39b8eb240cfa6f032b182dc277a P 3ec455833a2d41508940ed31beb2abce [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:12.729423 18002 raft_consensus.cc:504] T 3daef39b8eb240cfa6f032b182dc277a P 3ec455833a2d41508940ed31beb2abce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec455833a2d41508940ed31beb2abce" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42831 } } peers { permanent_uuid: "d614d61d0ef94dd897d302f86a3efe3e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38605 } } peers { permanent_uuid: "0cd69134625943ec86a2a7440139c0a0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33075 } }
I20260929 21:04:12.730022 18002 leader_election.cc:288] T 3daef39b8eb240cfa6f032b182dc277a P 3ec455833a2d41508940ed31beb2abce [CANDIDATE]: Term 1 election: Requested vote from peers d614d61d0ef94dd897d302f86a3efe3e (127.0.97.3:38605), 0cd69134625943ec86a2a7440139c0a0 (127.0.97.2:33075)
I20260929 21:04:12.730401 17933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3daef39b8eb240cfa6f032b182dc277a" candidate_uuid: "3ec455833a2d41508940ed31beb2abce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d614d61d0ef94dd897d302f86a3efe3e"
I20260929 21:04:12.730543 17800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3daef39b8eb240cfa6f032b182dc277a" candidate_uuid: "3ec455833a2d41508940ed31beb2abce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cd69134625943ec86a2a7440139c0a0"
I20260929 21:04:12.730615 17933 raft_consensus.cc:3037] T 3daef39b8eb240cfa6f032b182dc277a P d614d61d0ef94dd897d302f86a3efe3e [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:12.730762 17800 raft_consensus.cc:3037] T 3daef39b8eb240cfa6f032b182dc277a P 0cd69134625943ec86a2a7440139c0a0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:12.732738 17933 raft_consensus.cc:2454] T 3daef39b8eb240cfa6f032b182dc277a P d614d61d0ef94dd897d302f86a3efe3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ec455833a2d41508940ed31beb2abce in term 1.
I20260929 21:04:12.732924 17800 raft_consensus.cc:2454] T 3daef39b8eb240cfa6f032b182dc277a P 0cd69134625943ec86a2a7440139c0a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ec455833a2d41508940ed31beb2abce in term 1.
I20260929 21:04:12.733146 17599 leader_election.cc:302] T 3daef39b8eb240cfa6f032b182dc277a P 3ec455833a2d41508940ed31beb2abce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ec455833a2d41508940ed31beb2abce, d614d61d0ef94dd897d302f86a3efe3e; no voters:
I20260929 21:04:12.733459 18002 raft_consensus.cc:2781] T 3daef39b8eb240cfa6f032b182dc277a P 3ec455833a2d41508940ed31beb2abce [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:04:12.733875 18002 raft_consensus.cc:686] T 3daef39b8eb240cfa6f032b182dc277a P 3ec455833a2d41508940ed31beb2abce [term 1 LEADER]: Becoming Leader. State: Replica: 3ec455833a2d41508940ed31beb2abce, State: Running, Role: LEADER
I20260929 21:04:12.734305 18002 consensus_queue.cc:237] T 3daef39b8eb240cfa6f032b182dc277a P 3ec455833a2d41508940ed31beb2abce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec455833a2d41508940ed31beb2abce" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42831 } } peers { permanent_uuid: "d614d61d0ef94dd897d302f86a3efe3e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38605 } } peers { permanent_uuid: "0cd69134625943ec86a2a7440139c0a0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33075 } }
I20260929 21:04:12.738722 17522 catalog_manager.cc:5495] T 3daef39b8eb240cfa6f032b182dc277a P 3ec455833a2d41508940ed31beb2abce reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ec455833a2d41508940ed31beb2abce (127.0.97.1). New cstate: current_term: 1 leader_uuid: "3ec455833a2d41508940ed31beb2abce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec455833a2d41508940ed31beb2abce" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d614d61d0ef94dd897d302f86a3efe3e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cd69134625943ec86a2a7440139c0a0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33075 } health_report { overall_health: UNKNOWN } } }
I20260929 21:04:12.788698 17992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:12.783132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52604 (local address 127.0.97.1:42831)
0929 21:04:12.783362 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:12.783369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:12.783473 (+ 104us) server_negotiation.cc:407] Connection header received
0929 21:04:12.783736 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:12.783740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:12.783820 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:12.783974 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:04:12.784377 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:12.785148 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:12.786591 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:12.787001 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:12.787127 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:12.787611 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:04:12.788130 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:04:12.788297 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:12.788350 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 21:04:12.788430 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 21:04:12.796825 17800 raft_consensus.cc:1261] T 3daef39b8eb240cfa6f032b182dc277a P 0cd69134625943ec86a2a7440139c0a0 [term 1 FOLLOWER]: Refusing update from remote peer 3ec455833a2d41508940ed31beb2abce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:04:12.796825 17933 raft_consensus.cc:1261] T 3daef39b8eb240cfa6f032b182dc277a P d614d61d0ef94dd897d302f86a3efe3e [term 1 FOLLOWER]: Refusing update from remote peer 3ec455833a2d41508940ed31beb2abce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:04:12.797807 18002 consensus_queue.cc:1035] T 3daef39b8eb240cfa6f032b182dc277a P 3ec455833a2d41508940ed31beb2abce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cd69134625943ec86a2a7440139c0a0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:04:12.798115 18007 consensus_queue.cc:1035] T 3daef39b8eb240cfa6f032b182dc277a P 3ec455833a2d41508940ed31beb2abce [LEADER]: Connected to new peer: Peer: permanent_uuid: "d614d61d0ef94dd897d302f86a3efe3e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:04:12.814201 18010 mvcc.cc:205] Tried to move back new op lower bound from 7334772133045604352 to 7334772132803100672. Current Snapshot: MvccSnapshot[applied={T|T < 7334772133045604352}]
I20260929 21:04:12.815603 18011 mvcc.cc:205] Tried to move back new op lower bound from 7334772133045604352 to 7334772132803100672. Current Snapshot: MvccSnapshot[applied={T|T < 7334772133045604352}]
I20260929 21:04:12.817755 18012 mvcc.cc:205] Tried to move back new op lower bound from 7334772133045604352 to 7334772132803100672. Current Snapshot: MvccSnapshot[applied={T|T < 7334772133045604352}]
I20260929 21:04:12.833770 17521 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 21:04:12.834086 17521 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 21:04:12.837946 17521 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e336c942fab34021b1f401b0e48142b9: Sending DeleteTablet for 3 replicas of tablet 3daef39b8eb240cfa6f032b182dc277a
I20260929 21:04:12.839071 17646 tablet_service.cc:1514] Processing DeleteTablet for tablet 3daef39b8eb240cfa6f032b182dc277a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:04:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52600
I20260929 21:04:12.839226 17780 tablet_service.cc:1514] Processing DeleteTablet for tablet 3daef39b8eb240cfa6f032b182dc277a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:04:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43828
I20260929 21:04:12.839270 17913 tablet_service.cc:1514] Processing DeleteTablet for tablet 3daef39b8eb240cfa6f032b182dc277a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:04:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55398
I20260929 21:04:12.840668 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 17583
I20260929 21:04:12.848065 18021 tablet_replica.cc:330] T 3daef39b8eb240cfa6f032b182dc277a P 0cd69134625943ec86a2a7440139c0a0: stopping tablet replica
I20260929 21:04:12.848697 18021 raft_consensus.cc:2229] T 3daef39b8eb240cfa6f032b182dc277a P 0cd69134625943ec86a2a7440139c0a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:04:12.849367 18021 raft_consensus.cc:2258] T 3daef39b8eb240cfa6f032b182dc277a P 0cd69134625943ec86a2a7440139c0a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:04:12.850482 18022 tablet_replica.cc:330] T 3daef39b8eb240cfa6f032b182dc277a P d614d61d0ef94dd897d302f86a3efe3e: stopping tablet replica
I20260929 21:04:12.852286 18021 ts_tablet_manager.cc:1905] T 3daef39b8eb240cfa6f032b182dc277a P 0cd69134625943ec86a2a7440139c0a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:04:12.852938 18022 raft_consensus.cc:2229] T 3daef39b8eb240cfa6f032b182dc277a P d614d61d0ef94dd897d302f86a3efe3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:04:12.854213 18020 tablet_replica.cc:330] T 3daef39b8eb240cfa6f032b182dc277a P 3ec455833a2d41508940ed31beb2abce: stopping tablet replica
I20260929 21:04:12.854723 18020 raft_consensus.cc:2229] T 3daef39b8eb240cfa6f032b182dc277a P 3ec455833a2d41508940ed31beb2abce [term 1 LEADER]: Raft consensus shutting down.
I20260929 21:04:12.855440 18020 raft_consensus.cc:2258] T 3daef39b8eb240cfa6f032b182dc277a P 3ec455833a2d41508940ed31beb2abce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:04:12.855845 18022 raft_consensus.cc:2258] T 3daef39b8eb240cfa6f032b182dc277a P d614d61d0ef94dd897d302f86a3efe3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:04:12.857741 18020 ts_tablet_manager.cc:1905] T 3daef39b8eb240cfa6f032b182dc277a P 3ec455833a2d41508940ed31beb2abce: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 21:04:12 dist-test-slave-1dms krb5kdc[17477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 21:04:12.861052 18021 ts_tablet_manager.cc:1918] T 3daef39b8eb240cfa6f032b182dc277a P 0cd69134625943ec86a2a7440139c0a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:04:12.861227 18021 log.cc:1196] T 3daef39b8eb240cfa6f032b182dc277a P 0cd69134625943ec86a2a7440139c0a0: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-1/wal/wals/3daef39b8eb240cfa6f032b182dc277a
I20260929 21:04:12.866050 18020 ts_tablet_manager.cc:1918] T 3daef39b8eb240cfa6f032b182dc277a P 3ec455833a2d41508940ed31beb2abce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:04:12.866353 18020 log.cc:1196] T 3daef39b8eb240cfa6f032b182dc277a P 3ec455833a2d41508940ed31beb2abce: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-0/wal/wals/3daef39b8eb240cfa6f032b182dc277a
I20260929 21:04:12.866997 18020 ts_tablet_manager.cc:1939] T 3daef39b8eb240cfa6f032b182dc277a P 3ec455833a2d41508940ed31beb2abce: Deleting consensus metadata
I20260929 21:04:12.868271 18022 ts_tablet_manager.cc:1905] T 3daef39b8eb240cfa6f032b182dc277a P d614d61d0ef94dd897d302f86a3efe3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:04:12.868975 17509 catalog_manager.cc:4841] TS 3ec455833a2d41508940ed31beb2abce (127.0.97.1:42831): tablet 3daef39b8eb240cfa6f032b182dc277a (table test-table [id=ed36b4f837f544c488dfc258279c7ee9]) successfully deleted
I20260929 21:04:12.872220 17992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:12.842450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52614 (local address 127.0.97.1:42831)
0929 21:04:12.842689 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:12.842698 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:12.847655 (+ 4957us) server_negotiation.cc:407] Connection header received
0929 21:04:12.848545 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:12.848550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:12.848636 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:12.848725 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:12.850584 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:12.851801 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:12.853766 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:12.854520 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:12.854636 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:12.860662 (+ 6026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:12.860679 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:12.860698 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:04:12.860755 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:12.865639 (+ 4884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:12.869513 (+ 3874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:12.869522 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:12.869526 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:12.869710 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:12.870249 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:12.870254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:12.870258 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:12.870693 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:12.870995 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:12.871560 (+ 565us) server_negotiation.cc:299] Negotiation successful
0929 21:04:12.871830 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 21:04:12.873883 18021 ts_tablet_manager.cc:1939] T 3daef39b8eb240cfa6f032b182dc277a P 0cd69134625943ec86a2a7440139c0a0: Deleting consensus metadata
I20260929 21:04:12.873993 17706 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:04:12.875887 18022 ts_tablet_manager.cc:1918] T 3daef39b8eb240cfa6f032b182dc277a P d614d61d0ef94dd897d302f86a3efe3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:04:12.876123 18022 log.cc:1196] T 3daef39b8eb240cfa6f032b182dc277a P d614d61d0ef94dd897d302f86a3efe3e: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790715556572573-388-0/minicluster-data/ts-2/wal/wals/3daef39b8eb240cfa6f032b182dc277a
I20260929 21:04:12.876313 17506 catalog_manager.cc:4841] TS 0cd69134625943ec86a2a7440139c0a0 (127.0.97.2:33075): tablet 3daef39b8eb240cfa6f032b182dc277a (table test-table [id=ed36b4f837f544c488dfc258279c7ee9]) successfully deleted
I20260929 21:04:12.876747 18022 ts_tablet_manager.cc:1939] T 3daef39b8eb240cfa6f032b182dc277a P d614d61d0ef94dd897d302f86a3efe3e: Deleting consensus metadata
I20260929 21:04:12.878451 17506 catalog_manager.cc:4841] TS d614d61d0ef94dd897d302f86a3efe3e (127.0.97.3:38605): tablet 3daef39b8eb240cfa6f032b182dc277a (table test-table [id=ed36b4f837f544c488dfc258279c7ee9]) successfully deleted
I20260929 21:04:12.975847 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 17583
I20260929 21:04:12.993703 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 17717
Sep 29 21:04:12 dist-test-slave-1dms krb5kdc[17477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 21:04:13.008558 17995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:12.994539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43842 (local address 127.0.97.2:33075)
0929 21:04:12.994772 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:12.994777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:12.994794 (+ 17us) server_negotiation.cc:407] Connection header received
0929 21:04:12.994849 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:12.994853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:12.994924 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:12.995007 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:12.996213 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:12.997039 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:12.998294 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:12.998689 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:12.998783 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:13.003140 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:13.003153 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:13.003168 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:04:13.003204 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:13.006242 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:13.006937 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:13.006943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:13.006946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:13.007075 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:13.007369 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:13.007372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:13.007374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:13.007722 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:13.007940 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:13.008145 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 21:04:13.008298 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260929 21:04:13.009421 17840 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:04:13.097259 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 17717
I20260929 21:04:13.114228 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 17850
Sep 29 21:04:13 dist-test-slave-1dms krb5kdc[17477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 21:04:13.129490 17993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:13.114987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55404 (local address 127.0.97.3:38605)
0929 21:04:13.115181 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:13.115186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:13.115206 (+ 20us) server_negotiation.cc:407] Connection header received
0929 21:04:13.115394 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:13.115399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:13.115464 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:13.115564 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:13.116891 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:13.117596 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:13.119007 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:13.119402 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:13.119529 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:13.123985 (+ 4456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:13.124001 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:13.124016 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:04:13.124053 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:13.127001 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:13.127665 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:13.127672 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:13.127675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:13.127814 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:13.128152 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:13.128155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:13.128157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:13.128501 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:13.128722 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:13.129068 (+ 346us) server_negotiation.cc:299] Negotiation successful
0929 21:04:13.129226 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 21:04:13.130116 17973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:04:13.217641 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 17850
I20260929 21:04:13.234241 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 17490
I20260929 21:04:13.235328 17552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:04:13.346118 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 17490
2026-09-29T21:04:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4246 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T21:04:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:04:13Z Disabled control of system clock
I20260929 21:04:13.421509 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36599
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 21:04:13.740970 18031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:13.741240 18031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:13.741319 18031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:13.748236 18031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:04:13.748337 18031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:13.748375 18031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:13.748430 18031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:04:13.748474 18031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:04:13.757145 18031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36599
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:13.758306 18031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:13.759999 18031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:13.772185 18036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:13.772110 18037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:13.772143 18039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:13.778322 18031 server_base.cc:1034] running on GCE node
I20260929 21:04:13.779325 18031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:13.780808 18031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:13.782083 18031 hybrid_clock.cc:648] HybridClock initialized: now 1790715853782023 us; error 55 us; skew 500 ppm
I20260929 21:04:13.782433 18031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 21:04:13.784650 18031 webserver.cc:460] Webserver started at http://127.0.97.62:41171/ using document root <none> and password file <none>
I20260929 21:04:13.785377 18031 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:13.785516 18031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:13.785835 18031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:13.788699 18031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "71218175f21e4359b07f38c103261195"
format_stamp: "Formatted at 2026-09-29 21:04:13 on dist-test-slave-1dms"
server_key: "73dfe6893ca0aeeb16f55ebbfd82a624"
server_key_iv: "8628b586ccd3f6da5e50ddc0f4e1c705"
server_key_version: "encryptionkey@0"
I20260929 21:04:13.789532 18031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "71218175f21e4359b07f38c103261195"
format_stamp: "Formatted at 2026-09-29 21:04:13 on dist-test-slave-1dms"
server_key: "73dfe6893ca0aeeb16f55ebbfd82a624"
server_key_iv: "8628b586ccd3f6da5e50ddc0f4e1c705"
server_key_version: "encryptionkey@0"
I20260929 21:04:13.794464 18031 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 21:04:13.797842 18045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:13.799134 18031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 21:04:13.799314 18031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "71218175f21e4359b07f38c103261195"
format_stamp: "Formatted at 2026-09-29 21:04:13 on dist-test-slave-1dms"
server_key: "73dfe6893ca0aeeb16f55ebbfd82a624"
server_key_iv: "8628b586ccd3f6da5e50ddc0f4e1c705"
server_key_version: "encryptionkey@0"
I20260929 21:04:13.799523 18031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:13.815294 18031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:13.816110 18031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:13.816390 18031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:13.835057 18031 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36599
I20260929 21:04:13.835075 18096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36599 every 8 connection(s)
I20260929 21:04:13.836438 18031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:04:13.840251 18097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:04:13.841760 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 18031
I20260929 21:04:13.841950 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:04:13.842546 388 external_mini_cluster.cc:1428] Setting key 59f5cca3168a84c13cdf7491d7a88c0e
I20260929 21:04:13.853410 18097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71218175f21e4359b07f38c103261195: Bootstrap starting.
I20260929 21:04:13.855682 18100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:13.844470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34820 (local address 127.0.97.62:36599)
0929 21:04:13.845612 (+ 1142us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:13.845626 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:13.845672 (+ 46us) server_negotiation.cc:407] Connection header received
0929 21:04:13.847276 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:13.847298 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:13.847771 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:13.848179 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:13.848724 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:13.850014 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:13.851324 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:13.851984 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:13.852175 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:13.853446 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:13.853453 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:13.853466 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:13.854032 (+ 566us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:13.854111 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:13.854133 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:13.854261 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:13.854477 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 21:04:13.854781 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":444,"threads_started":1}
I20260929 21:04:13.856716 18097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71218175f21e4359b07f38c103261195: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:13.857774 18097 log.cc:824] T 00000000000000000000000000000000 P 71218175f21e4359b07f38c103261195: Log is configured to *not* fsync() on all Append() calls
I20260929 21:04:13.860374 18097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71218175f21e4359b07f38c103261195: No bootstrap required, opened a new log
I20260929 21:04:13.868039 18097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71218175f21e4359b07f38c103261195 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71218175f21e4359b07f38c103261195" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36599 } }
I20260929 21:04:13.868376 18097 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71218175f21e4359b07f38c103261195 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:13.868492 18097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71218175f21e4359b07f38c103261195 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71218175f21e4359b07f38c103261195, State: Initialized, Role: FOLLOWER
I20260929 21:04:13.869089 18097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71218175f21e4359b07f38c103261195 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71218175f21e4359b07f38c103261195" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36599 } }
I20260929 21:04:13.869325 18097 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71218175f21e4359b07f38c103261195 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:04:13.869470 18097 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71218175f21e4359b07f38c103261195 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:04:13.869679 18097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71218175f21e4359b07f38c103261195 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:13.871752 18097 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71218175f21e4359b07f38c103261195 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71218175f21e4359b07f38c103261195" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36599 } }
I20260929 21:04:13.872191 18097 leader_election.cc:302] T 00000000000000000000000000000000 P 71218175f21e4359b07f38c103261195 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71218175f21e4359b07f38c103261195; no voters:
I20260929 21:04:13.872717 18097 leader_election.cc:288] T 00000000000000000000000000000000 P 71218175f21e4359b07f38c103261195 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 21:04:13.872885 18102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71218175f21e4359b07f38c103261195 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:04:13.873605 18102 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71218175f21e4359b07f38c103261195 [term 1 LEADER]: Becoming Leader. State: Replica: 71218175f21e4359b07f38c103261195, State: Running, Role: LEADER
I20260929 21:04:13.874095 18102 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71218175f21e4359b07f38c103261195 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71218175f21e4359b07f38c103261195" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36599 } }
I20260929 21:04:13.874698 18097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71218175f21e4359b07f38c103261195 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:04:13.876816 18103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71218175f21e4359b07f38c103261195 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71218175f21e4359b07f38c103261195" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71218175f21e4359b07f38c103261195" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36599 } } }
I20260929 21:04:13.877307 18103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71218175f21e4359b07f38c103261195 [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:13.877656 18104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71218175f21e4359b07f38c103261195 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71218175f21e4359b07f38c103261195. Latest consensus state: current_term: 1 leader_uuid: "71218175f21e4359b07f38c103261195" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71218175f21e4359b07f38c103261195" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36599 } } }
I20260929 21:04:13.877972 18104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71218175f21e4359b07f38c103261195 [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:13.879829 18110 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:04:13.884732 18110 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:04:13.893255 18110 catalog_manager.cc:1347] Generated new cluster ID: b39d1f059cfb42b582ebf6f11a1cfda0
I20260929 21:04:13.893370 18110 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:04:13.903344 18110 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:04:13.904615 18110 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:04:13.911439 18110 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71218175f21e4359b07f38c103261195: Generated new TSK 0
I20260929 21:04:13.912168 18110 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 21:04:13.918619 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36599
--builtin_ntp_servers=127.0.97.20:42537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 21:04:14.249277 18121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:14.249581 18121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:14.249720 18121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:14.259529 18121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:14.259662 18121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:14.259882 18121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 21:04:14.272279 18121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:14.273638 18121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:14.275847 18121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:14.290360 18126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:14.290452 18129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:14.290405 18127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:14.291570 18121 server_base.cc:1034] running on GCE node
I20260929 21:04:14.292145 18121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:14.293125 18121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:14.294324 18121 hybrid_clock.cc:648] HybridClock initialized: now 1790715854294281 us; error 37 us; skew 500 ppm
I20260929 21:04:14.294633 18121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 21:04:14.296758 18121 webserver.cc:460] Webserver started at http://127.0.97.1:33173/ using document root <none> and password file <none>
I20260929 21:04:14.297462 18121 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:14.297576 18121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:14.297900 18121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:14.300678 18121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-0/data/instance:
uuid: "8e8fd7463654438385ea37ac1c381b8a"
format_stamp: "Formatted at 2026-09-29 21:04:14 on dist-test-slave-1dms"
server_key: "ae1efd15afd06701d18ab9d9430fa057"
server_key_iv: "705414bb6065e18b9cff70c296e863ce"
server_key_version: "encryptionkey@0"
I20260929 21:04:14.301527 18121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8e8fd7463654438385ea37ac1c381b8a"
format_stamp: "Formatted at 2026-09-29 21:04:14 on dist-test-slave-1dms"
server_key: "ae1efd15afd06701d18ab9d9430fa057"
server_key_iv: "705414bb6065e18b9cff70c296e863ce"
server_key_version: "encryptionkey@0"
I20260929 21:04:14.306675 18121 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 21:04:14.310048 18135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:14.311365 18121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 21:04:14.311537 18121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-0/wal
uuid: "8e8fd7463654438385ea37ac1c381b8a"
format_stamp: "Formatted at 2026-09-29 21:04:14 on dist-test-slave-1dms"
server_key: "ae1efd15afd06701d18ab9d9430fa057"
server_key_iv: "705414bb6065e18b9cff70c296e863ce"
server_key_version: "encryptionkey@0"
I20260929 21:04:14.311740 18121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:14.327373 18121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:14.328178 18121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:14.328455 18121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:14.329530 18121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:04:14.331007 18121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:04:14.331092 18121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:14.331172 18121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:04:14.331226 18121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:14.364537 18121 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38089
I20260929 21:04:14.364583 18247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38089 every 8 connection(s)
I20260929 21:04:14.365880 18121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
I20260929 21:04:14.375147 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 18121
I20260929 21:04:14.375319 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-0/wal/instance
I20260929 21:04:14.375851 388 external_mini_cluster.cc:1428] Setting key 8434d73f85fa4d2bfba093f369258a7d
I20260929 21:04:14.378660 18250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:14.368634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33235 (local address 127.0.97.62:36599)
0929 21:04:14.369114 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:14.369120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:14.370282 (+ 1162us) server_negotiation.cc:407] Connection header received
0929 21:04:14.371993 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:14.371999 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:14.372073 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:14.372170 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:14.373252 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:14.373962 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:14.375641 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:14.376238 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:14.376370 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:14.377819 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:14.377823 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:14.377826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:14.377843 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:14.377890 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:14.377903 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:14.378017 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:14.378238 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 21:04:14.378341 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":240,"threads_started":1}
I20260929 21:04:14.378922 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36599
--builtin_ntp_servers=127.0.97.20:42537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 21:04:14.379777 18251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:14.369275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36599 (local address 127.0.97.1:33235)
0929 21:04:14.370143 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:14.370178 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:14.371758 (+ 1580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:14.372360 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:14.372367 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:14.372448 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:04:14.373095 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:14.373114 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:14.374087 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:14.374090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:14.375494 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:14.375505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:14.376455 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:14.376472 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:14.376749 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:14.377476 (+ 727us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 21:04:14.377479 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:04:14.377525 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 21:04:14.377628 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:04:14.378016 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:04:14.378030 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:04:14.378040 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 21:04:14.378211 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 21:04:14.378584 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":276,"threads_started":1}
I20260929 21:04:14.381322 18248 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36599
I20260929 21:04:14.381620 18248 heartbeater.cc:464] Registering TS with master...
I20260929 21:04:14.382304 18248 heartbeater.cc:511] Master 127.0.97.62:36599 requested a full tablet report, sending...
I20260929 21:04:14.384373 18062 ts_manager.cc:194] Registered new tserver with Master: 8e8fd7463654438385ea37ac1c381b8a (127.0.97.1:38089)
I20260929 21:04:14.386582 18062 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:33235
W20260929 21:04:14.704970 18252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:14.705235 18252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:14.705382 18252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:14.712577 18252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:14.712677 18252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:14.712889 18252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 21:04:14.721944 18252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:14.723213 18252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:14.725028 18252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:14.737707 18257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:14.737782 18260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:14.737784 18258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:14.738659 18252 server_base.cc:1034] running on GCE node
I20260929 21:04:14.739256 18252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:14.740136 18252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:14.741357 18252 hybrid_clock.cc:648] HybridClock initialized: now 1790715854741308 us; error 43 us; skew 500 ppm
I20260929 21:04:14.741679 18252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 21:04:14.743772 18252 webserver.cc:460] Webserver started at http://127.0.97.2:44463/ using document root <none> and password file <none>
I20260929 21:04:14.744443 18252 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:14.744570 18252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:14.744942 18252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:14.747748 18252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-1/data/instance:
uuid: "f9498e1225384dbea04ea37b673f61ec"
format_stamp: "Formatted at 2026-09-29 21:04:14 on dist-test-slave-1dms"
server_key: "03dc6fd25299c92e49f6951f9886222a"
server_key_iv: "45085abc41733d61b79b632d01de3422"
server_key_version: "encryptionkey@0"
I20260929 21:04:14.748528 18252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f9498e1225384dbea04ea37b673f61ec"
format_stamp: "Formatted at 2026-09-29 21:04:14 on dist-test-slave-1dms"
server_key: "03dc6fd25299c92e49f6951f9886222a"
server_key_iv: "45085abc41733d61b79b632d01de3422"
server_key_version: "encryptionkey@0"
I20260929 21:04:14.753448 18252 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 21:04:14.756749 18266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:14.758097 18252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 21:04:14.758270 18252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-1/wal
uuid: "f9498e1225384dbea04ea37b673f61ec"
format_stamp: "Formatted at 2026-09-29 21:04:14 on dist-test-slave-1dms"
server_key: "03dc6fd25299c92e49f6951f9886222a"
server_key_iv: "45085abc41733d61b79b632d01de3422"
server_key_version: "encryptionkey@0"
I20260929 21:04:14.758513 18252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:14.790805 18252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:14.791664 18252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:14.791935 18252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:14.792986 18252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:04:14.794425 18252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:04:14.794552 18252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:14.794687 18252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:04:14.794773 18252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:14.827194 18252 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43563
I20260929 21:04:14.827279 18378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43563 every 8 connection(s)
I20260929 21:04:14.828493 18252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
I20260929 21:04:14.833675 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 18252
I20260929 21:04:14.833842 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-1/wal/instance
I20260929 21:04:14.834408 388 external_mini_cluster.cc:1428] Setting key 29f645f878b3e30463dcbf35b2ac0800
I20260929 21:04:14.837423 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36599
--builtin_ntp_servers=127.0.97.20:42537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 21:04:14.842346 18250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:14.831123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39171 (local address 127.0.97.62:36599)
0929 21:04:14.831345 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:14.831351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:14.832693 (+ 1342us) server_negotiation.cc:407] Connection header received
0929 21:04:14.835197 (+ 2504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:14.835202 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:14.835293 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:14.835404 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:14.837184 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:14.838040 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:14.839571 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:14.839990 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:14.840134 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:14.841507 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:14.841514 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:14.841524 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:14.841543 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:14.841576 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:14.841588 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:14.841686 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:14.841905 (+ 219us) server_negotiation.cc:299] Negotiation successful
0929 21:04:14.842004 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 21:04:14.843436 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:14.831666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36599 (local address 127.0.97.2:39171)
0929 21:04:14.832539 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:14.832580 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:14.834900 (+ 2320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:14.836246 (+ 1346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:14.836257 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:14.836376 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:04:14.837020 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:14.837038 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:14.838201 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:14.838211 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:14.839430 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:14.839442 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:14.840203 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:14.840209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:14.840435 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:14.841185 (+ 750us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 21:04:14.841188 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:04:14.841228 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 21:04:14.841307 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:04:14.841682 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:04:14.841694 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:04:14.841704 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 21:04:14.841879 (+ 175us) client_negotiation.cc:240] Negotiation successful
0929 21:04:14.842281 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":270,"threads_started":1}
I20260929 21:04:14.844854 18379 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36599
I20260929 21:04:14.845165 18379 heartbeater.cc:464] Registering TS with master...
I20260929 21:04:14.845896 18379 heartbeater.cc:511] Master 127.0.97.62:36599 requested a full tablet report, sending...
I20260929 21:04:14.847465 18062 ts_manager.cc:194] Registered new tserver with Master: f9498e1225384dbea04ea37b673f61ec (127.0.97.2:43563)
I20260929 21:04:14.848608 18062 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:39171
W20260929 21:04:15.160728 18382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:15.161082 18382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:15.161247 18382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:15.168701 18382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:15.168876 18382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:15.169055 18382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 21:04:15.178342 18382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:15.179685 18382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:15.181413 18382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:15.194044 18387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:15.194252 18390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:15.194382 18388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:15.195226 18382 server_base.cc:1034] running on GCE node
I20260929 21:04:15.195748 18382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:15.196614 18382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:15.197818 18382 hybrid_clock.cc:648] HybridClock initialized: now 1790715855197753 us; error 61 us; skew 500 ppm
I20260929 21:04:15.198098 18382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 21:04:15.200191 18382 webserver.cc:460] Webserver started at http://127.0.97.3:34893/ using document root <none> and password file <none>
I20260929 21:04:15.200804 18382 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:15.200949 18382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:15.201289 18382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:15.204298 18382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-2/data/instance:
uuid: "2e593decb15f447dba9ff83115e3fbb1"
format_stamp: "Formatted at 2026-09-29 21:04:15 on dist-test-slave-1dms"
server_key: "1620f69fa2106eefe30afba92d80a20a"
server_key_iv: "596eca1e23673d45a086c22cc80f5228"
server_key_version: "encryptionkey@0"
I20260929 21:04:15.205166 18382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-2/wal/instance:
uuid: "2e593decb15f447dba9ff83115e3fbb1"
format_stamp: "Formatted at 2026-09-29 21:04:15 on dist-test-slave-1dms"
server_key: "1620f69fa2106eefe30afba92d80a20a"
server_key_iv: "596eca1e23673d45a086c22cc80f5228"
server_key_version: "encryptionkey@0"
I20260929 21:04:15.210040 18382 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 21:04:15.213294 18396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:15.214542 18382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260929 21:04:15.214762 18382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-2/wal
uuid: "2e593decb15f447dba9ff83115e3fbb1"
format_stamp: "Formatted at 2026-09-29 21:04:15 on dist-test-slave-1dms"
server_key: "1620f69fa2106eefe30afba92d80a20a"
server_key_iv: "596eca1e23673d45a086c22cc80f5228"
server_key_version: "encryptionkey@0"
I20260929 21:04:15.214985 18382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:15.229054 18382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:15.229809 18382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:15.230083 18382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:15.230954 18382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:04:15.232337 18382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:04:15.232451 18382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:15.232618 18382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:04:15.232702 18382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:15.265304 18382 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41593
I20260929 21:04:15.265353 18508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41593 every 8 connection(s)
I20260929 21:04:15.266737 18382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
I20260929 21:04:15.274953 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 18382
I20260929 21:04:15.275131 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-2/wal/instance
I20260929 21:04:15.275592 388 external_mini_cluster.cc:1428] Setting key 3c0adcb5883a44c5c920d18307aa8820
I20260929 21:04:15.279399 18250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:15.269248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57671 (local address 127.0.97.62:36599)
0929 21:04:15.269474 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:15.269481 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:15.270943 (+ 1462us) server_negotiation.cc:407] Connection header received
0929 21:04:15.272677 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:15.272682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:15.272748 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:15.272882 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:15.273887 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:15.274703 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:15.276626 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:15.277212 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:15.277336 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:15.278725 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:15.278730 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:15.278733 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:15.278749 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:15.278780 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:15.278822 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:15.278909 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:15.279085 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 21:04:15.279161 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 21:04:15.280407 18511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:15.269784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36599 (local address 127.0.97.3:57671)
0929 21:04:15.270790 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:15.270831 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:15.272434 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:15.273091 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:15.273099 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:15.273187 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:04:15.273738 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:15.273753 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:15.274879 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:15.274883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:15.276493 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:15.276505 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:15.277428 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:15.277436 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:15.277666 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:15.278387 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 21:04:15.278390 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:04:15.278427 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 21:04:15.278506 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:04:15.278915 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:04:15.278923 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:04:15.278928 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 21:04:15.279064 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 21:04:15.279341 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":286,"threads_started":1}
I20260929 21:04:15.281838 18509 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36599
I20260929 21:04:15.282147 18509 heartbeater.cc:464] Registering TS with master...
I20260929 21:04:15.282860 18509 heartbeater.cc:511] Master 127.0.97.62:36599 requested a full tablet report, sending...
I20260929 21:04:15.284191 18062 ts_manager.cc:194] Registered new tserver with Master: 2e593decb15f447dba9ff83115e3fbb1 (127.0.97.3:41593)
I20260929 21:04:15.285018 18062 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:57671
I20260929 21:04:15.290342 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 21:04:15.299743 18250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:15.294191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34834 (local address 127.0.97.62:36599)
0929 21:04:15.294382 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:15.294390 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:15.294786 (+ 396us) server_negotiation.cc:407] Connection header received
0929 21:04:15.294910 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:15.294914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:15.294980 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:15.295059 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:15.295504 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:15.296198 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:15.297621 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:15.298005 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:15.298101 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:15.299129 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:15.299133 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:15.299136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:15.299152 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:15.299179 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:15.299191 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:15.299255 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:15.299362 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 21:04:15.299438 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 21:04:15.307587 18062 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 21:04:15.314306 18062 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 21:04:15.339054 18521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:15.326305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38089 (local address 127.0.0.1:56620)
0929 21:04:15.327079 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:15.327127 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:15.327312 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:15.328363 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:15.328375 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:15.328421 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:04:15.328913 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:15.328943 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:15.331497 (+ 2554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:15.331506 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:15.334704 (+ 3198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:15.334727 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:15.337820 (+ 3093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:15.337830 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:15.338040 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:15.338123 (+ 83us) client_negotiation.cc:769] Sending connection context
0929 21:04:15.338288 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 21:04:15.338574 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":357,"threads_started":1}
I20260929 21:04:15.340291 18522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:15.326529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56620 (local address 127.0.97.1:38089)
0929 21:04:15.327139 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:15.327152 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:15.327213 (+ 61us) server_negotiation.cc:407] Connection header received
0929 21:04:15.327528 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:15.327538 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:15.327802 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:15.327992 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:04:15.329196 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:15.331288 (+ 2092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:15.335022 (+ 3734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:15.337148 (+ 2126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:15.337320 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:15.337665 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:15.339747 (+ 2082us) server_negotiation.cc:299] Negotiation successful
0929 21:04:15.339952 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":313,"threads_started":1}
I20260929 21:04:15.344887 18183 tablet_service.cc:1467] Processing CreateTablet for tablet b0e1c1b1dc3e4697ac49430d1ae89015 (DEFAULT_TABLE table=test-table [id=8b575332acd5499abef83ed93226e1b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:04:15.346314 18183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0e1c1b1dc3e4697ac49430d1ae89015. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:04:15.348417 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:15.327474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41593 (local address 127.0.0.1:47928)
0929 21:04:15.328018 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:15.328034 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:15.328142 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:15.342462 (+ 14320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:15.342466 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:15.342493 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:04:15.342810 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:15.342820 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:15.344112 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:15.344128 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:15.346280 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:15.346293 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:15.347613 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:15.347619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:15.347797 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:15.347836 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 21:04:15.347928 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 21:04:15.348065 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":273,"threads_started":1}
I20260929 21:04:15.348469 18524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:15.327770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47928 (local address 127.0.97.3:41593)
0929 21:04:15.340193 (+ 12423us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:15.340203 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:15.340242 (+ 39us) server_negotiation.cc:407] Connection header received
0929 21:04:15.340349 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:15.340360 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:15.340659 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:15.342469 (+ 1810us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:04:15.342985 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:15.343973 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:15.346551 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:15.347416 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:15.347563 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:15.347845 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:15.347963 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 21:04:15.348113 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12235,"thread_start_us":12079,"threads_started":1}
I20260929 21:04:15.351186 18525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:15.328083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43563 (local address 127.0.0.1:41240)
0929 21:04:15.335095 (+ 7012us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:15.335116 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:15.335233 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:15.341887 (+ 6654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:15.341891 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:15.341914 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:04:15.342170 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:15.342177 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:15.345965 (+ 3788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:15.345970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:15.348031 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:15.348051 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:15.350446 (+ 2395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:15.350452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:15.350617 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:15.350655 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 21:04:15.350758 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 21:04:15.350881 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6895,"thread_start_us":6745,"threads_started":1}
I20260929 21:04:15.351274 18526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:15.328738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41240 (local address 127.0.97.2:43563)
0929 21:04:15.341293 (+ 12555us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:15.341302 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:15.341332 (+ 30us) server_negotiation.cc:407] Connection header received
0929 21:04:15.341437 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:15.341447 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:15.341689 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:15.341887 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:04:15.344182 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:15.345823 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:15.349015 (+ 3192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:15.350258 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:15.350394 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:15.350627 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:15.350804 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 21:04:15.350950 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12376,"thread_start_us":13355,"threads_started":1}
I20260929 21:04:15.352613 18444 tablet_service.cc:1467] Processing CreateTablet for tablet b0e1c1b1dc3e4697ac49430d1ae89015 (DEFAULT_TABLE table=test-table [id=8b575332acd5499abef83ed93226e1b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:04:15.354513 18444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0e1c1b1dc3e4697ac49430d1ae89015. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:04:15.354337 18314 tablet_service.cc:1467] Processing CreateTablet for tablet b0e1c1b1dc3e4697ac49430d1ae89015 (DEFAULT_TABLE table=test-table [id=8b575332acd5499abef83ed93226e1b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:04:15.355242 18314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0e1c1b1dc3e4697ac49430d1ae89015. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:04:15.360536 18527 tablet_bootstrap.cc:492] T b0e1c1b1dc3e4697ac49430d1ae89015 P 8e8fd7463654438385ea37ac1c381b8a: Bootstrap starting.
I20260929 21:04:15.364377 18527 tablet_bootstrap.cc:654] T b0e1c1b1dc3e4697ac49430d1ae89015 P 8e8fd7463654438385ea37ac1c381b8a: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:15.366353 18527 log.cc:824] T b0e1c1b1dc3e4697ac49430d1ae89015 P 8e8fd7463654438385ea37ac1c381b8a: Log is configured to *not* fsync() on all Append() calls
I20260929 21:04:15.374296 18529 tablet_bootstrap.cc:492] T b0e1c1b1dc3e4697ac49430d1ae89015 P f9498e1225384dbea04ea37b673f61ec: Bootstrap starting.
I20260929 21:04:15.375598 18530 tablet_bootstrap.cc:492] T b0e1c1b1dc3e4697ac49430d1ae89015 P 2e593decb15f447dba9ff83115e3fbb1: Bootstrap starting.
I20260929 21:04:15.378774 18527 tablet_bootstrap.cc:492] T b0e1c1b1dc3e4697ac49430d1ae89015 P 8e8fd7463654438385ea37ac1c381b8a: No bootstrap required, opened a new log
I20260929 21:04:15.379069 18527 ts_tablet_manager.cc:1397] T b0e1c1b1dc3e4697ac49430d1ae89015 P 8e8fd7463654438385ea37ac1c381b8a: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.004s
I20260929 21:04:15.380285 18529 tablet_bootstrap.cc:654] T b0e1c1b1dc3e4697ac49430d1ae89015 P f9498e1225384dbea04ea37b673f61ec: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:15.380683 18530 tablet_bootstrap.cc:654] T b0e1c1b1dc3e4697ac49430d1ae89015 P 2e593decb15f447dba9ff83115e3fbb1: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:15.381534 18529 log.cc:824] T b0e1c1b1dc3e4697ac49430d1ae89015 P f9498e1225384dbea04ea37b673f61ec: Log is configured to *not* fsync() on all Append() calls
I20260929 21:04:15.381654 18530 log.cc:824] T b0e1c1b1dc3e4697ac49430d1ae89015 P 2e593decb15f447dba9ff83115e3fbb1: Log is configured to *not* fsync() on all Append() calls
I20260929 21:04:15.384635 18529 tablet_bootstrap.cc:492] T b0e1c1b1dc3e4697ac49430d1ae89015 P f9498e1225384dbea04ea37b673f61ec: No bootstrap required, opened a new log
I20260929 21:04:15.384920 18529 ts_tablet_manager.cc:1397] T b0e1c1b1dc3e4697ac49430d1ae89015 P f9498e1225384dbea04ea37b673f61ec: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260929 21:04:15.387573 18530 tablet_bootstrap.cc:492] T b0e1c1b1dc3e4697ac49430d1ae89015 P 2e593decb15f447dba9ff83115e3fbb1: No bootstrap required, opened a new log
I20260929 21:04:15.387830 18530 ts_tablet_manager.cc:1397] T b0e1c1b1dc3e4697ac49430d1ae89015 P 2e593decb15f447dba9ff83115e3fbb1: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.003s
I20260929 21:04:15.389114 18527 raft_consensus.cc:348] T b0e1c1b1dc3e4697ac49430d1ae89015 P 8e8fd7463654438385ea37ac1c381b8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e8fd7463654438385ea37ac1c381b8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38089 } } peers { permanent_uuid: "2e593decb15f447dba9ff83115e3fbb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41593 } } peers { permanent_uuid: "f9498e1225384dbea04ea37b673f61ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43563 } }
I20260929 21:04:15.389561 18527 raft_consensus.cc:374] T b0e1c1b1dc3e4697ac49430d1ae89015 P 8e8fd7463654438385ea37ac1c381b8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:15.389664 18527 raft_consensus.cc:729] T b0e1c1b1dc3e4697ac49430d1ae89015 P 8e8fd7463654438385ea37ac1c381b8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e8fd7463654438385ea37ac1c381b8a, State: Initialized, Role: FOLLOWER
I20260929 21:04:15.390044 18248 heartbeater.cc:503] Master 127.0.97.62:36599 was elected leader, sending a full tablet report...
I20260929 21:04:15.390383 18527 consensus_queue.cc:260] T b0e1c1b1dc3e4697ac49430d1ae89015 P 8e8fd7463654438385ea37ac1c381b8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e8fd7463654438385ea37ac1c381b8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38089 } } peers { permanent_uuid: "2e593decb15f447dba9ff83115e3fbb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41593 } } peers { permanent_uuid: "f9498e1225384dbea04ea37b673f61ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43563 } }
I20260929 21:04:15.395934 18527 ts_tablet_manager.cc:1428] T b0e1c1b1dc3e4697ac49430d1ae89015 P 8e8fd7463654438385ea37ac1c381b8a: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260929 21:04:15.396406 18529 raft_consensus.cc:348] T b0e1c1b1dc3e4697ac49430d1ae89015 P f9498e1225384dbea04ea37b673f61ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e8fd7463654438385ea37ac1c381b8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38089 } } peers { permanent_uuid: "2e593decb15f447dba9ff83115e3fbb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41593 } } peers { permanent_uuid: "f9498e1225384dbea04ea37b673f61ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43563 } }
I20260929 21:04:15.396797 18529 raft_consensus.cc:374] T b0e1c1b1dc3e4697ac49430d1ae89015 P f9498e1225384dbea04ea37b673f61ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:15.396926 18529 raft_consensus.cc:729] T b0e1c1b1dc3e4697ac49430d1ae89015 P f9498e1225384dbea04ea37b673f61ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9498e1225384dbea04ea37b673f61ec, State: Initialized, Role: FOLLOWER
I20260929 21:04:15.397560 18529 consensus_queue.cc:260] T b0e1c1b1dc3e4697ac49430d1ae89015 P f9498e1225384dbea04ea37b673f61ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e8fd7463654438385ea37ac1c381b8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38089 } } peers { permanent_uuid: "2e593decb15f447dba9ff83115e3fbb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41593 } } peers { permanent_uuid: "f9498e1225384dbea04ea37b673f61ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43563 } }
I20260929 21:04:15.397884 18530 raft_consensus.cc:348] T b0e1c1b1dc3e4697ac49430d1ae89015 P 2e593decb15f447dba9ff83115e3fbb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e8fd7463654438385ea37ac1c381b8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38089 } } peers { permanent_uuid: "2e593decb15f447dba9ff83115e3fbb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41593 } } peers { permanent_uuid: "f9498e1225384dbea04ea37b673f61ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43563 } }
I20260929 21:04:15.398411 18530 raft_consensus.cc:374] T b0e1c1b1dc3e4697ac49430d1ae89015 P 2e593decb15f447dba9ff83115e3fbb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:15.398552 18530 raft_consensus.cc:729] T b0e1c1b1dc3e4697ac49430d1ae89015 P 2e593decb15f447dba9ff83115e3fbb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e593decb15f447dba9ff83115e3fbb1, State: Initialized, Role: FOLLOWER
I20260929 21:04:15.398615 18379 heartbeater.cc:503] Master 127.0.97.62:36599 was elected leader, sending a full tablet report...
I20260929 21:04:15.399250 18529 ts_tablet_manager.cc:1428] T b0e1c1b1dc3e4697ac49430d1ae89015 P f9498e1225384dbea04ea37b673f61ec: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260929 21:04:15.399271 18530 consensus_queue.cc:260] T b0e1c1b1dc3e4697ac49430d1ae89015 P 2e593decb15f447dba9ff83115e3fbb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e8fd7463654438385ea37ac1c381b8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38089 } } peers { permanent_uuid: "2e593decb15f447dba9ff83115e3fbb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41593 } } peers { permanent_uuid: "f9498e1225384dbea04ea37b673f61ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43563 } }
I20260929 21:04:15.400301 18509 heartbeater.cc:503] Master 127.0.97.62:36599 was elected leader, sending a full tablet report...
I20260929 21:04:15.401077 18530 ts_tablet_manager.cc:1428] T b0e1c1b1dc3e4697ac49430d1ae89015 P 2e593decb15f447dba9ff83115e3fbb1: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260929 21:04:15.496073 18534 raft_consensus.cc:482] T b0e1c1b1dc3e4697ac49430d1ae89015 P f9498e1225384dbea04ea37b673f61ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 21:04:15.496384 18534 raft_consensus.cc:504] T b0e1c1b1dc3e4697ac49430d1ae89015 P f9498e1225384dbea04ea37b673f61ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e8fd7463654438385ea37ac1c381b8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38089 } } peers { permanent_uuid: "2e593decb15f447dba9ff83115e3fbb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41593 } } peers { permanent_uuid: "f9498e1225384dbea04ea37b673f61ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43563 } }
I20260929 21:04:15.498081 18534 leader_election.cc:288] T b0e1c1b1dc3e4697ac49430d1ae89015 P f9498e1225384dbea04ea37b673f61ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e8fd7463654438385ea37ac1c381b8a (127.0.97.1:38089), 2e593decb15f447dba9ff83115e3fbb1 (127.0.97.3:41593)
I20260929 21:04:15.504585 18522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:15.498568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33347 (local address 127.0.97.1:38089)
0929 21:04:15.498816 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:15.498823 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:15.499267 (+ 444us) server_negotiation.cc:407] Connection header received
0929 21:04:15.499600 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:15.499606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:15.499706 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:15.499759 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:04:15.500374 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:15.501145 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:15.502920 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:15.503721 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:15.503853 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:15.503945 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:15.504215 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 21:04:15.504332 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 21:04:15.504673 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:15.498266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38089 (local address 127.0.97.2:33347)
0929 21:04:15.499165 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:15.499184 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:15.499405 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:15.499923 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:15.499927 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:15.499948 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:04:15.500235 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:15.500242 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:15.501265 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:15.501273 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:15.502733 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:15.502745 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:15.503889 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:15.503895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:15.504068 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:15.504091 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 21:04:15.504189 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 21:04:15.504309 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"mutex_wait_us":1,"thread_start_us":515,"threads_started":1}
I20260929 21:04:15.505203 18524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:15.499306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35137 (local address 127.0.97.3:41593)
0929 21:04:15.499574 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:15.499581 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:15.499870 (+ 289us) server_negotiation.cc:407] Connection header received
0929 21:04:15.500072 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:15.500078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:15.500144 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:15.500237 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:04:15.500755 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:15.501539 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:15.503339 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:15.504249 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:15.504375 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:15.504484 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:15.504800 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 21:04:15.504951 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 21:04:15.505841 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:15.499201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41593 (local address 127.0.97.2:35137)
0929 21:04:15.499757 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:15.499771 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:15.499920 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:15.500249 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:15.500254 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:15.500275 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:04:15.500609 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:15.500616 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:15.501676 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:15.501681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:15.503179 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:15.503191 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:15.504412 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:15.504418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:15.504584 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:15.504610 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 21:04:15.505376 (+ 766us) client_negotiation.cc:240] Negotiation successful
0929 21:04:15.505512 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":19,"thread_start_us":294,"threads_started":1}
I20260929 21:04:15.505846 18203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0e1c1b1dc3e4697ac49430d1ae89015" candidate_uuid: "f9498e1225384dbea04ea37b673f61ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e8fd7463654438385ea37ac1c381b8a" is_pre_election: true
I20260929 21:04:15.506398 18203 raft_consensus.cc:2454] T b0e1c1b1dc3e4697ac49430d1ae89015 P 8e8fd7463654438385ea37ac1c381b8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9498e1225384dbea04ea37b673f61ec in term 0.
I20260929 21:04:15.506711 18464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0e1c1b1dc3e4697ac49430d1ae89015" candidate_uuid: "f9498e1225384dbea04ea37b673f61ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e593decb15f447dba9ff83115e3fbb1" is_pre_election: true
I20260929 21:04:15.507041 18269 leader_election.cc:302] T b0e1c1b1dc3e4697ac49430d1ae89015 P f9498e1225384dbea04ea37b673f61ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e8fd7463654438385ea37ac1c381b8a, f9498e1225384dbea04ea37b673f61ec; no voters:
I20260929 21:04:15.507238 18464 raft_consensus.cc:2454] T b0e1c1b1dc3e4697ac49430d1ae89015 P 2e593decb15f447dba9ff83115e3fbb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9498e1225384dbea04ea37b673f61ec in term 0.
I20260929 21:04:15.507431 18534 raft_consensus.cc:2781] T b0e1c1b1dc3e4697ac49430d1ae89015 P f9498e1225384dbea04ea37b673f61ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 21:04:15.507571 18534 raft_consensus.cc:482] T b0e1c1b1dc3e4697ac49430d1ae89015 P f9498e1225384dbea04ea37b673f61ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 21:04:15.507702 18534 raft_consensus.cc:3037] T b0e1c1b1dc3e4697ac49430d1ae89015 P f9498e1225384dbea04ea37b673f61ec [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:15.509963 18534 raft_consensus.cc:504] T b0e1c1b1dc3e4697ac49430d1ae89015 P f9498e1225384dbea04ea37b673f61ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e8fd7463654438385ea37ac1c381b8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38089 } } peers { permanent_uuid: "2e593decb15f447dba9ff83115e3fbb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41593 } } peers { permanent_uuid: "f9498e1225384dbea04ea37b673f61ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43563 } }
I20260929 21:04:15.510512 18534 leader_election.cc:288] T b0e1c1b1dc3e4697ac49430d1ae89015 P f9498e1225384dbea04ea37b673f61ec [CANDIDATE]: Term 1 election: Requested vote from peers 8e8fd7463654438385ea37ac1c381b8a (127.0.97.1:38089), 2e593decb15f447dba9ff83115e3fbb1 (127.0.97.3:41593)
I20260929 21:04:15.510897 18203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0e1c1b1dc3e4697ac49430d1ae89015" candidate_uuid: "f9498e1225384dbea04ea37b673f61ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e8fd7463654438385ea37ac1c381b8a"
I20260929 21:04:15.511036 18464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0e1c1b1dc3e4697ac49430d1ae89015" candidate_uuid: "f9498e1225384dbea04ea37b673f61ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e593decb15f447dba9ff83115e3fbb1"
I20260929 21:04:15.511096 18203 raft_consensus.cc:3037] T b0e1c1b1dc3e4697ac49430d1ae89015 P 8e8fd7463654438385ea37ac1c381b8a [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:15.511252 18464 raft_consensus.cc:3037] T b0e1c1b1dc3e4697ac49430d1ae89015 P 2e593decb15f447dba9ff83115e3fbb1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:15.513325 18203 raft_consensus.cc:2454] T b0e1c1b1dc3e4697ac49430d1ae89015 P 8e8fd7463654438385ea37ac1c381b8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9498e1225384dbea04ea37b673f61ec in term 1.
I20260929 21:04:15.513558 18464 raft_consensus.cc:2454] T b0e1c1b1dc3e4697ac49430d1ae89015 P 2e593decb15f447dba9ff83115e3fbb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9498e1225384dbea04ea37b673f61ec in term 1.
I20260929 21:04:15.513703 18269 leader_election.cc:302] T b0e1c1b1dc3e4697ac49430d1ae89015 P f9498e1225384dbea04ea37b673f61ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e8fd7463654438385ea37ac1c381b8a, f9498e1225384dbea04ea37b673f61ec; no voters:
I20260929 21:04:15.513988 18534 raft_consensus.cc:2781] T b0e1c1b1dc3e4697ac49430d1ae89015 P f9498e1225384dbea04ea37b673f61ec [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:04:15.514441 18534 raft_consensus.cc:686] T b0e1c1b1dc3e4697ac49430d1ae89015 P f9498e1225384dbea04ea37b673f61ec [term 1 LEADER]: Becoming Leader. State: Replica: f9498e1225384dbea04ea37b673f61ec, State: Running, Role: LEADER
I20260929 21:04:15.514938 18534 consensus_queue.cc:237] T b0e1c1b1dc3e4697ac49430d1ae89015 P f9498e1225384dbea04ea37b673f61ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e8fd7463654438385ea37ac1c381b8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38089 } } peers { permanent_uuid: "2e593decb15f447dba9ff83115e3fbb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41593 } } peers { permanent_uuid: "f9498e1225384dbea04ea37b673f61ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43563 } }
W20260929 21:04:15.518402 18510 tablet.cc:2369] T b0e1c1b1dc3e4697ac49430d1ae89015 P 2e593decb15f447dba9ff83115e3fbb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 21:04:15.519399 18061 catalog_manager.cc:5495] T b0e1c1b1dc3e4697ac49430d1ae89015 P f9498e1225384dbea04ea37b673f61ec reported cstate change: term changed from 0 to 1, leader changed from <none> to f9498e1225384dbea04ea37b673f61ec (127.0.97.2). New cstate: current_term: 1 leader_uuid: "f9498e1225384dbea04ea37b673f61ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e8fd7463654438385ea37ac1c381b8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e593decb15f447dba9ff83115e3fbb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9498e1225384dbea04ea37b673f61ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43563 } health_report { overall_health: HEALTHY } } }
I20260929 21:04:15.537784 18526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:15.532397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41254 (local address 127.0.97.2:43563)
0929 21:04:15.532555 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:15.532559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:15.532574 (+ 15us) server_negotiation.cc:407] Connection header received
0929 21:04:15.532652 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:15.532655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:15.532725 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:15.532892 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:04:15.533451 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:15.534479 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:15.535805 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:15.536195 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:15.536285 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:15.536711 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:04:15.537248 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:04:15.537361 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:15.537445 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 21:04:15.537512 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 21:04:15.545205 18464 raft_consensus.cc:1261] T b0e1c1b1dc3e4697ac49430d1ae89015 P 2e593decb15f447dba9ff83115e3fbb1 [term 1 FOLLOWER]: Refusing update from remote peer f9498e1225384dbea04ea37b673f61ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:04:15.545284 18203 raft_consensus.cc:1261] T b0e1c1b1dc3e4697ac49430d1ae89015 P 8e8fd7463654438385ea37ac1c381b8a [term 1 FOLLOWER]: Refusing update from remote peer f9498e1225384dbea04ea37b673f61ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:04:15.546144 18538 consensus_queue.cc:1035] T b0e1c1b1dc3e4697ac49430d1ae89015 P f9498e1225384dbea04ea37b673f61ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e593decb15f447dba9ff83115e3fbb1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:04:15.546520 18534 consensus_queue.cc:1035] T b0e1c1b1dc3e4697ac49430d1ae89015 P f9498e1225384dbea04ea37b673f61ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e8fd7463654438385ea37ac1c381b8a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:04:15.557967 18543 mvcc.cc:205] Tried to move back new op lower bound from 7334772144304214016 to 7334772144193204224. Current Snapshot: MvccSnapshot[applied={T|T < 7334772144304214016}]
I20260929 21:04:15.571997 18061 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 21:04:15.572285 18061 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 21:04:15.575956 18061 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 71218175f21e4359b07f38c103261195: Sending DeleteTablet for 3 replicas of tablet b0e1c1b1dc3e4697ac49430d1ae89015
I20260929 21:04:15.577040 18183 tablet_service.cc:1514] Processing DeleteTablet for tablet b0e1c1b1dc3e4697ac49430d1ae89015 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:04:15 UTC) from {username='slave'} at 127.0.0.1:56620
I20260929 21:04:15.577339 18314 tablet_service.cc:1514] Processing DeleteTablet for tablet b0e1c1b1dc3e4697ac49430d1ae89015 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:04:15 UTC) from {username='slave'} at 127.0.0.1:41240
I20260929 21:04:15.577509 18444 tablet_service.cc:1514] Processing DeleteTablet for tablet b0e1c1b1dc3e4697ac49430d1ae89015 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:04:15 UTC) from {username='slave'} at 127.0.0.1:47928
I20260929 21:04:15.579180 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 18121
I20260929 21:04:15.589473 18553 tablet_replica.cc:330] T b0e1c1b1dc3e4697ac49430d1ae89015 P 2e593decb15f447dba9ff83115e3fbb1: stopping tablet replica
I20260929 21:04:15.589915 18551 tablet_replica.cc:330] T b0e1c1b1dc3e4697ac49430d1ae89015 P 8e8fd7463654438385ea37ac1c381b8a: stopping tablet replica
I20260929 21:04:15.590011 18553 raft_consensus.cc:2229] T b0e1c1b1dc3e4697ac49430d1ae89015 P 2e593decb15f447dba9ff83115e3fbb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:04:15.590478 18551 raft_consensus.cc:2229] T b0e1c1b1dc3e4697ac49430d1ae89015 P 8e8fd7463654438385ea37ac1c381b8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:04:15.590577 18553 raft_consensus.cc:2258] T b0e1c1b1dc3e4697ac49430d1ae89015 P 2e593decb15f447dba9ff83115e3fbb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:04:15.591076 18551 raft_consensus.cc:2258] T b0e1c1b1dc3e4697ac49430d1ae89015 P 8e8fd7463654438385ea37ac1c381b8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:04:15.593526 18553 ts_tablet_manager.cc:1905] T b0e1c1b1dc3e4697ac49430d1ae89015 P 2e593decb15f447dba9ff83115e3fbb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:04:15.597101 18552 tablet_replica.cc:330] T b0e1c1b1dc3e4697ac49430d1ae89015 P f9498e1225384dbea04ea37b673f61ec: stopping tablet replica
I20260929 21:04:15.597445 18552 raft_consensus.cc:2229] T b0e1c1b1dc3e4697ac49430d1ae89015 P f9498e1225384dbea04ea37b673f61ec [term 1 LEADER]: Raft consensus shutting down.
I20260929 21:04:15.598042 18552 raft_consensus.cc:2258] T b0e1c1b1dc3e4697ac49430d1ae89015 P f9498e1225384dbea04ea37b673f61ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:04:15.600037 18551 ts_tablet_manager.cc:1905] T b0e1c1b1dc3e4697ac49430d1ae89015 P 8e8fd7463654438385ea37ac1c381b8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:04:15.600171 18552 ts_tablet_manager.cc:1905] T b0e1c1b1dc3e4697ac49430d1ae89015 P f9498e1225384dbea04ea37b673f61ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:04:15.600392 18522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:15.582259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56636 (local address 127.0.97.1:38089)
0929 21:04:15.582488 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:15.582497 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:15.585582 (+ 3085us) server_negotiation.cc:407] Connection header received
0929 21:04:15.585777 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:15.585786 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:15.585924 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:15.586064 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:15.590071 (+ 4007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:15.591643 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:15.594623 (+ 2980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:15.595507 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:15.595630 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:15.598734 (+ 3104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:15.598748 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:15.598765 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:15.599514 (+ 749us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:15.599579 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:15.599603 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:15.599714 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:15.599936 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 21:04:15.600054 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 21:04:15.600860 18553 ts_tablet_manager.cc:1918] T b0e1c1b1dc3e4697ac49430d1ae89015 P 2e593decb15f447dba9ff83115e3fbb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:04:15.601049 18553 log.cc:1196] T b0e1c1b1dc3e4697ac49430d1ae89015 P 2e593decb15f447dba9ff83115e3fbb1: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-2/wal/wals/b0e1c1b1dc3e4697ac49430d1ae89015
I20260929 21:04:15.601552 18243 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:04:15.601584 18553 ts_tablet_manager.cc:1939] T b0e1c1b1dc3e4697ac49430d1ae89015 P 2e593decb15f447dba9ff83115e3fbb1: Deleting consensus metadata
I20260929 21:04:15.603694 18048 catalog_manager.cc:4841] TS 2e593decb15f447dba9ff83115e3fbb1 (127.0.97.3:41593): tablet b0e1c1b1dc3e4697ac49430d1ae89015 (table test-table [id=8b575332acd5499abef83ed93226e1b2]) successfully deleted
I20260929 21:04:15.609277 18552 ts_tablet_manager.cc:1918] T b0e1c1b1dc3e4697ac49430d1ae89015 P f9498e1225384dbea04ea37b673f61ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:04:15.609450 18552 log.cc:1196] T b0e1c1b1dc3e4697ac49430d1ae89015 P f9498e1225384dbea04ea37b673f61ec: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790715556572573-388-0/minicluster-data/ts-1/wal/wals/b0e1c1b1dc3e4697ac49430d1ae89015
I20260929 21:04:15.610008 18552 ts_tablet_manager.cc:1939] T b0e1c1b1dc3e4697ac49430d1ae89015 P f9498e1225384dbea04ea37b673f61ec: Deleting consensus metadata
I20260929 21:04:15.611811 18048 catalog_manager.cc:4841] TS f9498e1225384dbea04ea37b673f61ec (127.0.97.2:43563): tablet b0e1c1b1dc3e4697ac49430d1ae89015 (table test-table [id=8b575332acd5499abef83ed93226e1b2]) successfully deleted
I20260929 21:04:15.701627 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 18121
I20260929 21:04:15.719790 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 18252
W20260929 21:04:15.719821 18048 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:38089 (error 108)
I20260929 21:04:15.722239 18525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:15.721812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38089 (local address 127.0.0.1:56648)
0929 21:04:15.721976 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:15.722095 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 21:04:15.722551 18048 catalog_manager.cc:4584] TS 8e8fd7463654438385ea37ac1c381b8a (127.0.97.1:38089): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b0e1c1b1dc3e4697ac49430d1ae89015: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38089: connect: Connection refused (error 111)
I20260929 21:04:15.728011 18526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:15.721090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41262 (local address 127.0.97.2:43563)
0929 21:04:15.721266 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:15.721270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:15.721290 (+ 20us) server_negotiation.cc:407] Connection header received
0929 21:04:15.721345 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:15.721348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:15.721423 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:15.721519 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:15.722012 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:15.723187 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:15.724453 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:15.725172 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:15.725316 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:15.726312 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:15.726323 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:15.726342 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:15.727174 (+ 832us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:15.727234 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:15.727259 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:15.727375 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:15.727524 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 21:04:15.727623 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 21:04:15.728899 18374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:04:15.776250 18525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:15.775698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38089 (local address 127.0.0.1:56650)
0929 21:04:15.775916 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:15.776049 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 21:04:15.777521 18525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:15.777121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38089 (local address 127.0.0.1:56662)
0929 21:04:15.777282 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:15.777372 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 21:04:15.838183 18525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:15.837636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38089 (local address 127.0.0.1:56676)
0929 21:04:15.837873 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:15.837984 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 21:04:15.839548 18525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:15.839116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38089 (local address 127.0.0.1:56688)
0929 21:04:15.839276 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:15.839392 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 21:04:15.840739 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 18252
I20260929 21:04:15.859179 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 18382
I20260929 21:04:15.865667 18524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:15.859868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47938 (local address 127.0.97.3:41593)
0929 21:04:15.860022 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:15.860026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:15.860062 (+ 36us) server_negotiation.cc:407] Connection header received
0929 21:04:15.860231 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:15.860235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:15.860325 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:15.860410 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:15.860783 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:15.861537 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:15.862839 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:15.863199 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:15.863286 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:15.864350 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:15.864358 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:15.864372 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:15.865030 (+ 658us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:15.865078 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:15.865105 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:15.865193 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:15.865329 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 21:04:15.865415 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 21:04:15.866297 18504 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:04:15.920639 18525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:15.920113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38089 (local address 127.0.0.1:56702)
0929 21:04:15.920360 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:15.920478 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 21:04:15.921830 18525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:15.921406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38089 (local address 127.0.0.1:56706)
0929 21:04:15.921637 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:15.921713 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260929 21:04:15.954912 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 18382
I20260929 21:04:15.971473 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 18031
I20260929 21:04:15.972633 18092 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:04:16.081337 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 18031
2026-09-29T21:04:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2733 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 21:04:16 dist-test-slave-1dms krb5kdc[18562](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 21:04:16 dist-test-slave-1dms krb5kdc[18562](info): set up 2 sockets
Sep 29 21:04:16 dist-test-slave-1dms krb5kdc[18562](info): commencing operation
krb5kdc: starting...
W20260929 21:04:17.177107 388 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 21:04:17 dist-test-slave-1dms krb5kdc[18562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T21:04:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:04:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 21:04:17.410228 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46119
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46119
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:04:17.731437 18575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:17.731741 18575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:17.731832 18575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:17.739274 18575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:04:17.739390 18575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:17.739445 18575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:17.739508 18575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:04:17.739552 18575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:04:17.749370 18575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:17.750625 18575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:17.751942 18575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:17.763213 18581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:17.763175 18580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:17.763242 18583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:17.764263 18575 server_base.cc:1034] running on GCE node
I20260929 21:04:17.765084 18575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:17.766465 18575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:17.767668 18575 hybrid_clock.cc:648] HybridClock initialized: now 1790715857767608 us; error 50 us; skew 500 ppm
I20260929 21:04:17.768023 18575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:04:17 dist-test-slave-1dms krb5kdc[18562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:04:17.771133 18575 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 21:04:17.772696 18575 webserver.cc:460] Webserver started at http://127.0.97.62:38849/ using document root <none> and password file <none>
I20260929 21:04:17.773506 18575 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:17.773656 18575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:17.773978 18575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:17.777272 18575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "a110ee0adb634e6fab484e2f0fc9024b"
format_stamp: "Formatted at 2026-09-29 21:04:17 on dist-test-slave-1dms"
server_key: "305fb33c9c60109812201dba75af37f0"
server_key_iv: "2aa941fa7547b0882f7edee8970c70a9"
server_key_version: "encryptionkey@0"
I20260929 21:04:17.778086 18575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "a110ee0adb634e6fab484e2f0fc9024b"
format_stamp: "Formatted at 2026-09-29 21:04:17 on dist-test-slave-1dms"
server_key: "305fb33c9c60109812201dba75af37f0"
server_key_iv: "2aa941fa7547b0882f7edee8970c70a9"
server_key_version: "encryptionkey@0"
I20260929 21:04:17.783386 18575 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 21:04:17.786994 18590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:17.788313 18575 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 21:04:17.788529 18575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "a110ee0adb634e6fab484e2f0fc9024b"
format_stamp: "Formatted at 2026-09-29 21:04:17 on dist-test-slave-1dms"
server_key: "305fb33c9c60109812201dba75af37f0"
server_key_iv: "2aa941fa7547b0882f7edee8970c70a9"
server_key_version: "encryptionkey@0"
I20260929 21:04:17.788743 18575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:17.807858 18575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:17.811931 18575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:17.812232 18575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:17.831667 18575 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46119
I20260929 21:04:17.831691 18641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46119 every 8 connection(s)
I20260929 21:04:17.833112 18575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:04:17.836675 18642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:04:17.842451 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 18575
I20260929 21:04:17.842645 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:04:17.843070 388 external_mini_cluster.cc:1428] Setting key 1a759916b64a3ab2380a37905f851dda
I20260929 21:04:17.847193 18642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a110ee0adb634e6fab484e2f0fc9024b: Bootstrap starting.
I20260929 21:04:17.851019 18642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a110ee0adb634e6fab484e2f0fc9024b: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:17.852283 18642 log.cc:824] T 00000000000000000000000000000000 P a110ee0adb634e6fab484e2f0fc9024b: Log is configured to *not* fsync() on all Append() calls
I20260929 21:04:17.855069 18642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a110ee0adb634e6fab484e2f0fc9024b: No bootstrap required, opened a new log
Sep 29 21:04:17 dist-test-slave-1dms krb5kdc[18562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:04:17.864763 18642 raft_consensus.cc:348] T 00000000000000000000000000000000 P a110ee0adb634e6fab484e2f0fc9024b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a110ee0adb634e6fab484e2f0fc9024b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46119 } }
I20260929 21:04:17.865240 18642 raft_consensus.cc:374] T 00000000000000000000000000000000 P a110ee0adb634e6fab484e2f0fc9024b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:17.865351 18642 raft_consensus.cc:729] T 00000000000000000000000000000000 P a110ee0adb634e6fab484e2f0fc9024b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a110ee0adb634e6fab484e2f0fc9024b, State: Initialized, Role: FOLLOWER
I20260929 21:04:17.865998 18642 consensus_queue.cc:260] T 00000000000000000000000000000000 P a110ee0adb634e6fab484e2f0fc9024b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a110ee0adb634e6fab484e2f0fc9024b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46119 } }
I20260929 21:04:17.866315 18642 raft_consensus.cc:388] T 00000000000000000000000000000000 P a110ee0adb634e6fab484e2f0fc9024b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:04:17.866483 18642 raft_consensus.cc:482] T 00000000000000000000000000000000 P a110ee0adb634e6fab484e2f0fc9024b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:04:17.866675 18642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a110ee0adb634e6fab484e2f0fc9024b [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:17.868253 18645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:17.845202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39448 (local address 127.0.97.62:46119)
0929 21:04:17.846117 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:17.846132 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:17.846586 (+ 454us) server_negotiation.cc:407] Connection header received
0929 21:04:17.849183 (+ 2597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:17.849211 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:17.849713 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:17.850113 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:17.851417 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:17.853276 (+ 1859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:17.854583 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:17.855598 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:17.855789 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:17.859831 (+ 4042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:17.859841 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:17.859865 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:04:17.859917 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:17.863974 (+ 4057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:17.864662 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:17.864671 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:17.864678 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:17.864905 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:17.865263 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:17.865267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:17.865271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:17.865979 (+ 708us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:17.866287 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:17.866772 (+ 485us) server_negotiation.cc:299] Negotiation successful
0929 21:04:17.867194 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":428,"threads_started":1}
I20260929 21:04:17.869215 18642 raft_consensus.cc:504] T 00000000000000000000000000000000 P a110ee0adb634e6fab484e2f0fc9024b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a110ee0adb634e6fab484e2f0fc9024b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46119 } }
I20260929 21:04:17.869786 18642 leader_election.cc:302] T 00000000000000000000000000000000 P a110ee0adb634e6fab484e2f0fc9024b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a110ee0adb634e6fab484e2f0fc9024b; no voters:
I20260929 21:04:17.870431 18642 leader_election.cc:288] T 00000000000000000000000000000000 P a110ee0adb634e6fab484e2f0fc9024b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 21:04:17.870543 18647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a110ee0adb634e6fab484e2f0fc9024b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:04:17.872426 18642 sys_catalog.cc:564] T 00000000000000000000000000000000 P a110ee0adb634e6fab484e2f0fc9024b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:04:17.874534 18647 raft_consensus.cc:686] T 00000000000000000000000000000000 P a110ee0adb634e6fab484e2f0fc9024b [term 1 LEADER]: Becoming Leader. State: Replica: a110ee0adb634e6fab484e2f0fc9024b, State: Running, Role: LEADER
I20260929 21:04:17.875242 18647 consensus_queue.cc:237] T 00000000000000000000000000000000 P a110ee0adb634e6fab484e2f0fc9024b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a110ee0adb634e6fab484e2f0fc9024b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46119 } }
I20260929 21:04:17.878211 18648 sys_catalog.cc:455] T 00000000000000000000000000000000 P a110ee0adb634e6fab484e2f0fc9024b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a110ee0adb634e6fab484e2f0fc9024b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a110ee0adb634e6fab484e2f0fc9024b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46119 } } }
I20260929 21:04:17.878609 18648 sys_catalog.cc:458] T 00000000000000000000000000000000 P a110ee0adb634e6fab484e2f0fc9024b [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:17.877912 18649 sys_catalog.cc:455] T 00000000000000000000000000000000 P a110ee0adb634e6fab484e2f0fc9024b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a110ee0adb634e6fab484e2f0fc9024b. Latest consensus state: current_term: 1 leader_uuid: "a110ee0adb634e6fab484e2f0fc9024b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a110ee0adb634e6fab484e2f0fc9024b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46119 } } }
I20260929 21:04:17.879493 18649 sys_catalog.cc:458] T 00000000000000000000000000000000 P a110ee0adb634e6fab484e2f0fc9024b [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:17.879580 18661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:04:17.889110 18661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:04:17.897059 18661 catalog_manager.cc:1347] Generated new cluster ID: d8636135c4be4c42aaaa197bbde84e2e
I20260929 21:04:17.897197 18661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:04:17.915529 18661 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:04:17.916908 18661 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:04:17.930954 18661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a110ee0adb634e6fab484e2f0fc9024b: Generated new TSK 0
I20260929 21:04:17.931847 18661 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 21:04:17.999451 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46119
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:04:18.322224 18669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:18.322527 18669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:18.322688 18669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:18.329752 18669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:18.329852 18669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:18.330049 18669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 21:04:18.339389 18669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:18.340590 18669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:18.341877 18669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:18.355590 18674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:18.355634 18675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:18.355676 18677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:18.356279 18669 server_base.cc:1034] running on GCE node
I20260929 21:04:18.356962 18669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:18.357826 18669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:18.359019 18669 hybrid_clock.cc:648] HybridClock initialized: now 1790715858358976 us; error 38 us; skew 500 ppm
I20260929 21:04:18.359349 18669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:04:18 dist-test-slave-1dms krb5kdc[18562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:04:18.362361 18669 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 21:04:18.363652 18669 webserver.cc:460] Webserver started at http://127.0.97.1:44003/ using document root <none> and password file <none>
I20260929 21:04:18.364267 18669 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:18.364358 18669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:18.364598 18669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:18.367381 18669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-0/data/instance:
uuid: "1071939dd7de49afade31b646d72bcf1"
format_stamp: "Formatted at 2026-09-29 21:04:18 on dist-test-slave-1dms"
server_key: "0abdbf2b3451cd6235fc9c932106e43d"
server_key_iv: "a37556067f3caf05b28ba436d104a43a"
server_key_version: "encryptionkey@0"
I20260929 21:04:18.368075 18669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1071939dd7de49afade31b646d72bcf1"
format_stamp: "Formatted at 2026-09-29 21:04:18 on dist-test-slave-1dms"
server_key: "0abdbf2b3451cd6235fc9c932106e43d"
server_key_iv: "a37556067f3caf05b28ba436d104a43a"
server_key_version: "encryptionkey@0"
I20260929 21:04:18.372910 18669 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 21:04:18.377049 18684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:18.378360 18669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 21:04:18.378532 18669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-0/wal
uuid: "1071939dd7de49afade31b646d72bcf1"
format_stamp: "Formatted at 2026-09-29 21:04:18 on dist-test-slave-1dms"
server_key: "0abdbf2b3451cd6235fc9c932106e43d"
server_key_iv: "a37556067f3caf05b28ba436d104a43a"
server_key_version: "encryptionkey@0"
I20260929 21:04:18.378742 18669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:18.403836 18669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:18.407881 18669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:18.408169 18669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:18.409179 18669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:04:18.410485 18669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:04:18.410588 18669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:18.410727 18669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:04:18.410795 18669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:18.444635 18669 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36879
I20260929 21:04:18.444679 18796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36879 every 8 connection(s)
I20260929 21:04:18.445917 18669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
I20260929 21:04:18.452550 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 18669
I20260929 21:04:18.452744 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-0/wal/instance
I20260929 21:04:18.453325 388 external_mini_cluster.cc:1428] Setting key 209795011e7be7481fd6b6b90b2cce17
Sep 29 21:04:18 dist-test-slave-1dms krb5kdc[18562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:04:18.468007 18799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:18.448598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58197 (local address 127.0.97.62:46119)
0929 21:04:18.449087 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:18.449091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:18.450057 (+ 966us) server_negotiation.cc:407] Connection header received
0929 21:04:18.451812 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:18.451816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:18.451883 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:18.451980 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:18.454044 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:18.455106 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:18.457598 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:18.458026 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:18.458134 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:18.462308 (+ 4174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:18.462312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:18.462316 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:04:18.462348 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:18.464983 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:18.465727 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:18.465738 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:18.465740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:18.465855 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:18.466270 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:18.466275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:18.466278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:18.466557 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:18.466709 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:18.467412 (+ 703us) server_negotiation.cc:299] Negotiation successful
0929 21:04:18.467641 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":257,"threads_started":1}
I20260929 21:04:18.469157 18800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:18.449220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46119 (local address 127.0.97.1:58197)
0929 21:04:18.449889 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:18.449936 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:18.451565 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:18.452200 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:18.452210 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:18.453027 (+ 817us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:04:18.453848 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:18.453873 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:18.455266 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:18.455270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:18.457432 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:18.457449 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:18.458268 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:18.458278 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:18.458604 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:18.459363 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:04:18.459397 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:04:18.462069 (+ 2672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:04:18.465125 (+ 3056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:04:18.465134 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:04:18.465152 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:04:18.465598 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:04:18.465963 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:04:18.465968 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:04:18.465972 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:04:18.466156 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:04:18.466694 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:04:18.466702 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:04:18.467039 (+ 337us) client_negotiation.cc:769] Sending connection context
0929 21:04:18.467303 (+ 264us) client_negotiation.cc:240] Negotiation successful
0929 21:04:18.467852 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":261,"threads_started":1}
I20260929 21:04:18.470775 18797 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46119
I20260929 21:04:18.471105 18797 heartbeater.cc:464] Registering TS with master...
I20260929 21:04:18.471982 18797 heartbeater.cc:511] Master 127.0.97.62:46119 requested a full tablet report, sending...
I20260929 21:04:18.474556 18607 ts_manager.cc:194] Registered new tserver with Master: 1071939dd7de49afade31b646d72bcf1 (127.0.97.1:36879)
I20260929 21:04:18.476912 18607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58197
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 21:04:18.509941 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46119
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:04:18.831972 18803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:18.832233 18803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:18.832382 18803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:18.839530 18803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:18.839634 18803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:18.839808 18803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 21:04:18.849489 18803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:18.850682 18803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:18.851977 18803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:18.864562 18808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:18.864615 18811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:18.864622 18809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:18.865154 18803 server_base.cc:1034] running on GCE node
I20260929 21:04:18.865774 18803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:18.866734 18803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:18.867951 18803 hybrid_clock.cc:648] HybridClock initialized: now 1790715858867901 us; error 45 us; skew 500 ppm
I20260929 21:04:18.868270 18803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:04:18 dist-test-slave-1dms krb5kdc[18562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:04:18.871315 18803 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 21:04:18.872867 18803 webserver.cc:460] Webserver started at http://127.0.97.2:36183/ using document root <none> and password file <none>
I20260929 21:04:18.873594 18803 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:18.873728 18803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:18.874013 18803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:18.877080 18803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-1/data/instance:
uuid: "4914fb8984884648912625357ebdfe4b"
format_stamp: "Formatted at 2026-09-29 21:04:18 on dist-test-slave-1dms"
server_key: "60d53826c33cd8b5151798a188153ad4"
server_key_iv: "b869754e3bdd4246895a59c1ccc88000"
server_key_version: "encryptionkey@0"
I20260929 21:04:18.877905 18803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4914fb8984884648912625357ebdfe4b"
format_stamp: "Formatted at 2026-09-29 21:04:18 on dist-test-slave-1dms"
server_key: "60d53826c33cd8b5151798a188153ad4"
server_key_iv: "b869754e3bdd4246895a59c1ccc88000"
server_key_version: "encryptionkey@0"
I20260929 21:04:18.882982 18803 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 21:04:18.886301 18818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:18.887622 18803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 21:04:18.887748 18803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-1/wal
uuid: "4914fb8984884648912625357ebdfe4b"
format_stamp: "Formatted at 2026-09-29 21:04:18 on dist-test-slave-1dms"
server_key: "60d53826c33cd8b5151798a188153ad4"
server_key_iv: "b869754e3bdd4246895a59c1ccc88000"
server_key_version: "encryptionkey@0"
I20260929 21:04:18.887909 18803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:18.913353 18803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:18.917245 18803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:18.917481 18803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:18.918382 18803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:04:18.919732 18803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:04:18.919803 18803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:18.919884 18803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:04:18.919929 18803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:18.952281 18803 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45425
I20260929 21:04:18.952327 18930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45425 every 8 connection(s)
I20260929 21:04:18.953620 18803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
I20260929 21:04:18.963260 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 18803
I20260929 21:04:18.963450 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-1/wal/instance
I20260929 21:04:18.963987 388 external_mini_cluster.cc:1428] Setting key 4aff120ce916f29f3f3db28ba23f10fe
Sep 29 21:04:18 dist-test-slave-1dms krb5kdc[18562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:04:18.973692 18799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:18.956065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42247 (local address 127.0.97.62:46119)
0929 21:04:18.956244 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:18.956248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:18.957457 (+ 1209us) server_negotiation.cc:407] Connection header received
0929 21:04:18.959216 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:18.959221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:18.959289 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:18.959379 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:18.960768 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:18.961563 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:18.963149 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:18.963663 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:18.963759 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:18.967485 (+ 3726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:18.967490 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:18.967493 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:04:18.967523 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:18.970114 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:18.971081 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:18.971089 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:18.971094 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:18.971290 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:18.971827 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:18.971832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:18.971835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:18.972174 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:18.972286 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:18.973167 (+ 881us) server_negotiation.cc:299] Negotiation successful
0929 21:04:18.973354 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 21:04:18.974870 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:18.956566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46119 (local address 127.0.97.2:42247)
0929 21:04:18.957302 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:18.957341 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:18.958959 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:18.959573 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:18.959581 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:18.960080 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:04:18.960620 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:18.960636 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:18.961707 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:18.961711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:18.962974 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:18.962996 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:18.963877 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:18.963885 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:18.964181 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:18.964963 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:04:18.964994 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:04:18.967274 (+ 2280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:04:18.970280 (+ 3006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:04:18.970292 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:04:18.970312 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:04:18.970941 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:04:18.971428 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:04:18.971435 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:04:18.971441 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:04:18.971699 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:04:18.972294 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:04:18.972305 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:04:18.972746 (+ 441us) client_negotiation.cc:769] Sending connection context
0929 21:04:18.973087 (+ 341us) client_negotiation.cc:240] Negotiation successful
0929 21:04:18.973658 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":312,"threads_started":1}
I20260929 21:04:18.976344 18931 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46119
I20260929 21:04:18.976629 18931 heartbeater.cc:464] Registering TS with master...
I20260929 21:04:18.977330 18931 heartbeater.cc:511] Master 127.0.97.62:46119 requested a full tablet report, sending...
I20260929 21:04:18.978880 18607 ts_manager.cc:194] Registered new tserver with Master: 4914fb8984884648912625357ebdfe4b (127.0.97.2:45425)
I20260929 21:04:18.979679 18607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42247
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 21:04:19.017786 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46119
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:04:19.334753 18936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:19.335072 18936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:19.335238 18936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:19.342392 18936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:19.342548 18936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:19.342715 18936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 21:04:19.352674 18936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:19.354243 18936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:19.355697 18936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:19.368434 18941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:19.368351 18944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:19.368403 18942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:19.369248 18936 server_base.cc:1034] running on GCE node
I20260929 21:04:19.369776 18936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:19.370616 18936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:19.371809 18936 hybrid_clock.cc:648] HybridClock initialized: now 1790715859371766 us; error 38 us; skew 500 ppm
I20260929 21:04:19.372114 18936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:04:19 dist-test-slave-1dms krb5kdc[18562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715859, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:04:19.381497 18936 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 21:04:19.383014 18936 webserver.cc:460] Webserver started at http://127.0.97.3:40075/ using document root <none> and password file <none>
I20260929 21:04:19.383770 18936 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:19.383903 18936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:19.384194 18936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:19.387240 18936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-2/data/instance:
uuid: "b3fbc329151a4176903c2fdc296eeaf3"
format_stamp: "Formatted at 2026-09-29 21:04:19 on dist-test-slave-1dms"
server_key: "fc5f1543a722aa7ccffa944f7d34697f"
server_key_iv: "9244f607c29d256b6b16a59f13b158d7"
server_key_version: "encryptionkey@0"
I20260929 21:04:19.387995 18936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b3fbc329151a4176903c2fdc296eeaf3"
format_stamp: "Formatted at 2026-09-29 21:04:19 on dist-test-slave-1dms"
server_key: "fc5f1543a722aa7ccffa944f7d34697f"
server_key_iv: "9244f607c29d256b6b16a59f13b158d7"
server_key_version: "encryptionkey@0"
I20260929 21:04:19.393352 18936 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 21:04:19.396574 18951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:19.397866 18936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:04:19.398031 18936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-2/wal
uuid: "b3fbc329151a4176903c2fdc296eeaf3"
format_stamp: "Formatted at 2026-09-29 21:04:19 on dist-test-slave-1dms"
server_key: "fc5f1543a722aa7ccffa944f7d34697f"
server_key_iv: "9244f607c29d256b6b16a59f13b158d7"
server_key_version: "encryptionkey@0"
I20260929 21:04:19.398219 18936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:19.411746 18936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:19.415931 18936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:19.416220 18936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:19.417200 18936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:04:19.418437 18936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:04:19.418534 18936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:19.418678 18936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:04:19.418752 18936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:19.451411 18936 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42829
I20260929 21:04:19.451459 19063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42829 every 8 connection(s)
I20260929 21:04:19.452610 18936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
I20260929 21:04:19.459743 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 18936
I20260929 21:04:19.459932 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-2/wal/instance
I20260929 21:04:19.460405 388 external_mini_cluster.cc:1428] Setting key d6753f698d088056e5d0be65571e4355
Sep 29 21:04:19 dist-test-slave-1dms krb5kdc[18562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715859, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:04:19.472138 18799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:19.455040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48119 (local address 127.0.97.62:46119)
0929 21:04:19.455247 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:19.455254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:19.456340 (+ 1086us) server_negotiation.cc:407] Connection header received
0929 21:04:19.458213 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:19.458219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:19.458306 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:19.458416 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:19.460135 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:19.461114 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:19.463102 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:19.463482 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:19.463575 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:19.466866 (+ 3291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:19.466871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:19.466874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:04:19.466906 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:19.469524 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:19.470255 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:19.470260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:19.470263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:19.470417 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:19.470782 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:19.470785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:19.470787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:19.471025 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:19.471138 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:19.471725 (+ 587us) server_negotiation.cc:299] Negotiation successful
0929 21:04:19.471885 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 21:04:19.473193 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:19.455504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46119 (local address 127.0.97.3:48119)
0929 21:04:19.456185 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:19.456222 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:19.457939 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:19.458653 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:19.458661 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:19.459164 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:04:19.459954 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:19.459980 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:19.461268 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:19.461274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:19.462944 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:19.462958 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:19.463677 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:19.463681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:19.463864 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:19.464466 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:04:19.464528 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:04:19.466695 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:04:19.469642 (+ 2947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:04:19.469649 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:04:19.469660 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:04:19.470149 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:04:19.470507 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:04:19.470510 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:04:19.470512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:04:19.470694 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:04:19.471121 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:04:19.471127 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:04:19.471417 (+ 290us) client_negotiation.cc:769] Sending connection context
0929 21:04:19.471650 (+ 233us) client_negotiation.cc:240] Negotiation successful
0929 21:04:19.472006 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":262,"threads_started":1}
I20260929 21:04:19.474733 19064 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46119
I20260929 21:04:19.475059 19064 heartbeater.cc:464] Registering TS with master...
I20260929 21:04:19.475996 19064 heartbeater.cc:511] Master 127.0.97.62:46119 requested a full tablet report, sending...
I20260929 21:04:19.477339 18606 ts_manager.cc:194] Registered new tserver with Master: b3fbc329151a4176903c2fdc296eeaf3 (127.0.97.3:42829)
I20260929 21:04:19.478192 18606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48119
I20260929 21:04:19.480563 18797 heartbeater.cc:503] Master 127.0.97.62:46119 was elected leader, sending a full tablet report...
I20260929 21:04:19.487514 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 21:04:19.504024 18799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:19.491488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38668 (local address 127.0.97.62:46119)
0929 21:04:19.491697 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:19.491702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:19.492098 (+ 396us) server_negotiation.cc:407] Connection header received
0929 21:04:19.492285 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:19.492293 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:19.492349 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:19.492423 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:19.494043 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:19.494748 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:19.495930 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:19.496314 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:19.496403 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:19.498804 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:19.498809 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:19.498812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:04:19.498842 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:19.501427 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:19.502135 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:19.502141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:19.502167 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:19.502284 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:19.502759 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:19.502762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:19.502764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:19.503044 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:19.503343 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:19.503668 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 21:04:19.503800 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 21:04:19.507786 18606 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 21:04:19.514601 18606 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 21:04:19.535946 19077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:19.525827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45425 (local address 127.0.0.1:37698)
0929 21:04:19.526865 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:19.526907 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:19.527057 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:19.528244 (+ 1187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:19.528254 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:19.528296 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:04:19.528784 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:19.528802 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:19.530744 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:19.530751 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:19.533137 (+ 2386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:19.533162 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:19.534972 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:19.534978 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:19.535181 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:19.535240 (+ 59us) client_negotiation.cc:769] Sending connection context
0929 21:04:19.535379 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 21:04:19.535570 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"thread_start_us":1820,"threads_started":1}
I20260929 21:04:19.535964 19078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:19.525967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37698 (local address 127.0.97.2:45425)
0929 21:04:19.527569 (+ 1602us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:19.527580 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:19.527608 (+ 28us) server_negotiation.cc:407] Connection header received
0929 21:04:19.527729 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:19.527740 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:19.527988 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:19.528924 (+ 936us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:04:19.529226 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:19.530584 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:19.533315 (+ 2731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:19.534775 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:19.534904 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:19.535081 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:19.535433 (+ 352us) server_negotiation.cc:299] Negotiation successful
0929 21:04:19.535620 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1441,"thread_start_us":1325,"threads_started":1}
I20260929 21:04:19.537098 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:19.525945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42829 (local address 127.0.0.1:48348)
0929 21:04:19.527923 (+ 1978us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:19.527946 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:19.528082 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:19.530573 (+ 2491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:19.530577 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:19.530602 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:04:19.531057 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:19.531070 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:19.532779 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:19.532784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:19.534464 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:19.534476 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:19.536314 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:19.536322 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:19.536485 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:19.536525 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 21:04:19.536591 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 21:04:19.536713 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1818,"mutex_wait_us":1,"thread_start_us":574,"threads_started":1}
I20260929 21:04:19.538213 19079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:19.526280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48348 (local address 127.0.97.3:42829)
0929 21:04:19.530016 (+ 3736us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:19.530024 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:19.530055 (+ 31us) server_negotiation.cc:407] Connection header received
0929 21:04:19.530141 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:19.530148 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:19.530360 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:19.530550 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:04:19.531243 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:19.532607 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:19.534617 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:19.535944 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:19.537417 (+ 1473us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:19.537640 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:19.537713 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 21:04:19.537860 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3572,"thread_start_us":3402,"threads_started":1}
I20260929 21:04:19.538811 19081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:19.527818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52538 (local address 127.0.97.1:36879)
0929 21:04:19.528784 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:19.528795 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:19.530038 (+ 1243us) server_negotiation.cc:407] Connection header received
0929 21:04:19.530152 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:19.530158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:19.530374 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:19.530550 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:04:19.531859 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:19.533407 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:19.536457 (+ 3050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:19.537766 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:19.537906 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:19.538120 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:19.538324 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 21:04:19.538477 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":750,"thread_start_us":558,"threads_started":1}
I20260929 21:04:19.538843 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:19.527261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36879 (local address 127.0.0.1:52538)
0929 21:04:19.529343 (+ 2082us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:19.529361 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:19.529468 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:19.531251 (+ 1783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:19.531255 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:19.531277 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:04:19.531687 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:19.531704 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:19.533550 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:19.533560 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:19.535514 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:19.535527 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:19.537951 (+ 2424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:19.537957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:19.538142 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:19.538176 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 21:04:19.538432 (+ 256us) client_negotiation.cc:240] Negotiation successful
0929 21:04:19.538547 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1970,"thread_start_us":1857,"threads_started":1}
I20260929 21:04:19.539994 18866 tablet_service.cc:1467] Processing CreateTablet for tablet 5c4fea01aa9042b6a2dcd099db5749f8 (DEFAULT_TABLE table=test-table [id=483cd24e677e483b8a6e1659a1156465]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:04:19.541783 18866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c4fea01aa9042b6a2dcd099db5749f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:04:19.541553 18999 tablet_service.cc:1467] Processing CreateTablet for tablet 5c4fea01aa9042b6a2dcd099db5749f8 (DEFAULT_TABLE table=test-table [id=483cd24e677e483b8a6e1659a1156465]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:04:19.542670 18999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c4fea01aa9042b6a2dcd099db5749f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:04:19.542546 18732 tablet_service.cc:1467] Processing CreateTablet for tablet 5c4fea01aa9042b6a2dcd099db5749f8 (DEFAULT_TABLE table=test-table [id=483cd24e677e483b8a6e1659a1156465]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:04:19.543804 18732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c4fea01aa9042b6a2dcd099db5749f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:04:19.553588 19082 tablet_bootstrap.cc:492] T 5c4fea01aa9042b6a2dcd099db5749f8 P b3fbc329151a4176903c2fdc296eeaf3: Bootstrap starting.
I20260929 21:04:19.556636 19082 tablet_bootstrap.cc:654] T 5c4fea01aa9042b6a2dcd099db5749f8 P b3fbc329151a4176903c2fdc296eeaf3: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:19.556825 19083 tablet_bootstrap.cc:492] T 5c4fea01aa9042b6a2dcd099db5749f8 P 4914fb8984884648912625357ebdfe4b: Bootstrap starting.
I20260929 21:04:19.558090 19082 log.cc:824] T 5c4fea01aa9042b6a2dcd099db5749f8 P b3fbc329151a4176903c2fdc296eeaf3: Log is configured to *not* fsync() on all Append() calls
I20260929 21:04:19.558320 19084 tablet_bootstrap.cc:492] T 5c4fea01aa9042b6a2dcd099db5749f8 P 1071939dd7de49afade31b646d72bcf1: Bootstrap starting.
I20260929 21:04:19.560200 19083 tablet_bootstrap.cc:654] T 5c4fea01aa9042b6a2dcd099db5749f8 P 4914fb8984884648912625357ebdfe4b: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:19.561345 19083 log.cc:824] T 5c4fea01aa9042b6a2dcd099db5749f8 P 4914fb8984884648912625357ebdfe4b: Log is configured to *not* fsync() on all Append() calls
I20260929 21:04:19.561779 19084 tablet_bootstrap.cc:654] T 5c4fea01aa9042b6a2dcd099db5749f8 P 1071939dd7de49afade31b646d72bcf1: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:19.562980 19084 log.cc:824] T 5c4fea01aa9042b6a2dcd099db5749f8 P 1071939dd7de49afade31b646d72bcf1: Log is configured to *not* fsync() on all Append() calls
I20260929 21:04:19.563155 19082 tablet_bootstrap.cc:492] T 5c4fea01aa9042b6a2dcd099db5749f8 P b3fbc329151a4176903c2fdc296eeaf3: No bootstrap required, opened a new log
I20260929 21:04:19.563428 19082 ts_tablet_manager.cc:1397] T 5c4fea01aa9042b6a2dcd099db5749f8 P b3fbc329151a4176903c2fdc296eeaf3: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 21:04:19.564504 19083 tablet_bootstrap.cc:492] T 5c4fea01aa9042b6a2dcd099db5749f8 P 4914fb8984884648912625357ebdfe4b: No bootstrap required, opened a new log
I20260929 21:04:19.564759 19083 ts_tablet_manager.cc:1397] T 5c4fea01aa9042b6a2dcd099db5749f8 P 4914fb8984884648912625357ebdfe4b: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 21:04:19.565578 19084 tablet_bootstrap.cc:492] T 5c4fea01aa9042b6a2dcd099db5749f8 P 1071939dd7de49afade31b646d72bcf1: No bootstrap required, opened a new log
I20260929 21:04:19.565821 19084 ts_tablet_manager.cc:1397] T 5c4fea01aa9042b6a2dcd099db5749f8 P 1071939dd7de49afade31b646d72bcf1: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 21:04:19.573386 19084 raft_consensus.cc:348] T 5c4fea01aa9042b6a2dcd099db5749f8 P 1071939dd7de49afade31b646d72bcf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4914fb8984884648912625357ebdfe4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45425 } } peers { permanent_uuid: "b3fbc329151a4176903c2fdc296eeaf3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42829 } } peers { permanent_uuid: "1071939dd7de49afade31b646d72bcf1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36879 } }
I20260929 21:04:19.573764 19084 raft_consensus.cc:374] T 5c4fea01aa9042b6a2dcd099db5749f8 P 1071939dd7de49afade31b646d72bcf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:19.573855 19084 raft_consensus.cc:729] T 5c4fea01aa9042b6a2dcd099db5749f8 P 1071939dd7de49afade31b646d72bcf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1071939dd7de49afade31b646d72bcf1, State: Initialized, Role: FOLLOWER
I20260929 21:04:19.574558 19084 consensus_queue.cc:260] T 5c4fea01aa9042b6a2dcd099db5749f8 P 1071939dd7de49afade31b646d72bcf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4914fb8984884648912625357ebdfe4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45425 } } peers { permanent_uuid: "b3fbc329151a4176903c2fdc296eeaf3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42829 } } peers { permanent_uuid: "1071939dd7de49afade31b646d72bcf1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36879 } }
I20260929 21:04:19.575222 19082 raft_consensus.cc:348] T 5c4fea01aa9042b6a2dcd099db5749f8 P b3fbc329151a4176903c2fdc296eeaf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4914fb8984884648912625357ebdfe4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45425 } } peers { permanent_uuid: "b3fbc329151a4176903c2fdc296eeaf3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42829 } } peers { permanent_uuid: "1071939dd7de49afade31b646d72bcf1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36879 } }
I20260929 21:04:19.575670 19082 raft_consensus.cc:374] T 5c4fea01aa9042b6a2dcd099db5749f8 P b3fbc329151a4176903c2fdc296eeaf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:19.575769 19082 raft_consensus.cc:729] T 5c4fea01aa9042b6a2dcd099db5749f8 P b3fbc329151a4176903c2fdc296eeaf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3fbc329151a4176903c2fdc296eeaf3, State: Initialized, Role: FOLLOWER
I20260929 21:04:19.575847 19083 raft_consensus.cc:348] T 5c4fea01aa9042b6a2dcd099db5749f8 P 4914fb8984884648912625357ebdfe4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4914fb8984884648912625357ebdfe4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45425 } } peers { permanent_uuid: "b3fbc329151a4176903c2fdc296eeaf3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42829 } } peers { permanent_uuid: "1071939dd7de49afade31b646d72bcf1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36879 } }
I20260929 21:04:19.576246 19083 raft_consensus.cc:374] T 5c4fea01aa9042b6a2dcd099db5749f8 P 4914fb8984884648912625357ebdfe4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:19.576335 19083 raft_consensus.cc:729] T 5c4fea01aa9042b6a2dcd099db5749f8 P 4914fb8984884648912625357ebdfe4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4914fb8984884648912625357ebdfe4b, State: Initialized, Role: FOLLOWER
I20260929 21:04:19.576467 19084 ts_tablet_manager.cc:1428] T 5c4fea01aa9042b6a2dcd099db5749f8 P 1071939dd7de49afade31b646d72bcf1: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260929 21:04:19.576500 19082 consensus_queue.cc:260] T 5c4fea01aa9042b6a2dcd099db5749f8 P b3fbc329151a4176903c2fdc296eeaf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4914fb8984884648912625357ebdfe4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45425 } } peers { permanent_uuid: "b3fbc329151a4176903c2fdc296eeaf3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42829 } } peers { permanent_uuid: "1071939dd7de49afade31b646d72bcf1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36879 } }
I20260929 21:04:19.577018 19083 consensus_queue.cc:260] T 5c4fea01aa9042b6a2dcd099db5749f8 P 4914fb8984884648912625357ebdfe4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4914fb8984884648912625357ebdfe4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45425 } } peers { permanent_uuid: "b3fbc329151a4176903c2fdc296eeaf3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42829 } } peers { permanent_uuid: "1071939dd7de49afade31b646d72bcf1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36879 } }
I20260929 21:04:19.578079 19064 heartbeater.cc:503] Master 127.0.97.62:46119 was elected leader, sending a full tablet report...
I20260929 21:04:19.579030 19082 ts_tablet_manager.cc:1428] T 5c4fea01aa9042b6a2dcd099db5749f8 P b3fbc329151a4176903c2fdc296eeaf3: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260929 21:04:19.580430 18931 heartbeater.cc:503] Master 127.0.97.62:46119 was elected leader, sending a full tablet report...
I20260929 21:04:19.581148 19083 ts_tablet_manager.cc:1428] T 5c4fea01aa9042b6a2dcd099db5749f8 P 4914fb8984884648912625357ebdfe4b: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
I20260929 21:04:19.590004 19088 raft_consensus.cc:482] T 5c4fea01aa9042b6a2dcd099db5749f8 P 1071939dd7de49afade31b646d72bcf1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 21:04:19.590287 19088 raft_consensus.cc:504] T 5c4fea01aa9042b6a2dcd099db5749f8 P 1071939dd7de49afade31b646d72bcf1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4914fb8984884648912625357ebdfe4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45425 } } peers { permanent_uuid: "b3fbc329151a4176903c2fdc296eeaf3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42829 } } peers { permanent_uuid: "1071939dd7de49afade31b646d72bcf1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36879 } }
I20260929 21:04:19.591955 19088 leader_election.cc:288] T 5c4fea01aa9042b6a2dcd099db5749f8 P 1071939dd7de49afade31b646d72bcf1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4914fb8984884648912625357ebdfe4b (127.0.97.2:45425), b3fbc329151a4176903c2fdc296eeaf3 (127.0.97.3:42829)
I20260929 21:04:19.599946 19079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:19.592626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49431 (local address 127.0.97.3:42829)
0929 21:04:19.592794 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:19.592799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:19.593359 (+ 560us) server_negotiation.cc:407] Connection header received
0929 21:04:19.593547 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:19.593554 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:19.593634 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:19.593700 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:04:19.594388 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:19.595171 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:19.598126 (+ 2955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:19.599112 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:19.599223 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:19.599297 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:19.599563 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 21:04:19.599692 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 21:04:19.600061 19092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:19.592354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42829 (local address 127.0.97.1:49431)
0929 21:04:19.593242 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:19.593256 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:19.593400 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:19.593892 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:19.593896 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:19.593918 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:04:19.594238 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:19.594245 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:19.595355 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:19.595364 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:19.597948 (+ 2584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:19.597967 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:19.599295 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:19.599300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:19.599406 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:19.599429 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 21:04:19.599538 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 21:04:19.599675 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"mutex_wait_us":174,"thread_start_us":580,"threads_started":1}
I20260929 21:04:19.600994 19019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c4fea01aa9042b6a2dcd099db5749f8" candidate_uuid: "1071939dd7de49afade31b646d72bcf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3fbc329151a4176903c2fdc296eeaf3" is_pre_election: true
I20260929 21:04:19.601208 19078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:19.592215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45599 (local address 127.0.97.2:45425)
0929 21:04:19.592461 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:19.592468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:19.593000 (+ 532us) server_negotiation.cc:407] Connection header received
0929 21:04:19.593269 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:19.593275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:19.593362 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:19.593445 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:04:19.594368 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:19.595173 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:19.599364 (+ 4191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:19.600250 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:19.600381 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:19.600486 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:19.600818 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 21:04:19.600960 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260929 21:04:19.601338 19091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:19.592153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45425 (local address 127.0.97.1:45599)
0929 21:04:19.592891 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:19.592916 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:19.593113 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:19.593484 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:19.593490 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:19.593518 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:04:19.593889 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:19.593899 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:19.595300 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:19.595308 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:19.597966 (+ 2658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:19.597976 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:19.600515 (+ 2539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:19.600523 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:19.600660 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:19.600683 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 21:04:19.600792 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 21:04:19.600948 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":127,"thread_start_us":367,"threads_started":1}
I20260929 21:04:19.601540 19019 raft_consensus.cc:2454] T 5c4fea01aa9042b6a2dcd099db5749f8 P b3fbc329151a4176903c2fdc296eeaf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1071939dd7de49afade31b646d72bcf1 in term 0.
I20260929 21:04:19.602267 18688 leader_election.cc:302] T 5c4fea01aa9042b6a2dcd099db5749f8 P 1071939dd7de49afade31b646d72bcf1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1071939dd7de49afade31b646d72bcf1, b3fbc329151a4176903c2fdc296eeaf3; no voters:
I20260929 21:04:19.602268 18886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c4fea01aa9042b6a2dcd099db5749f8" candidate_uuid: "1071939dd7de49afade31b646d72bcf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4914fb8984884648912625357ebdfe4b" is_pre_election: true
I20260929 21:04:19.602710 19088 raft_consensus.cc:2781] T 5c4fea01aa9042b6a2dcd099db5749f8 P 1071939dd7de49afade31b646d72bcf1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 21:04:19.602766 18886 raft_consensus.cc:2454] T 5c4fea01aa9042b6a2dcd099db5749f8 P 4914fb8984884648912625357ebdfe4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1071939dd7de49afade31b646d72bcf1 in term 0.
I20260929 21:04:19.602833 19088 raft_consensus.cc:482] T 5c4fea01aa9042b6a2dcd099db5749f8 P 1071939dd7de49afade31b646d72bcf1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 21:04:19.602954 19088 raft_consensus.cc:3037] T 5c4fea01aa9042b6a2dcd099db5749f8 P 1071939dd7de49afade31b646d72bcf1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:19.605572 19088 raft_consensus.cc:504] T 5c4fea01aa9042b6a2dcd099db5749f8 P 1071939dd7de49afade31b646d72bcf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4914fb8984884648912625357ebdfe4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45425 } } peers { permanent_uuid: "b3fbc329151a4176903c2fdc296eeaf3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42829 } } peers { permanent_uuid: "1071939dd7de49afade31b646d72bcf1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36879 } }
I20260929 21:04:19.606187 19088 leader_election.cc:288] T 5c4fea01aa9042b6a2dcd099db5749f8 P 1071939dd7de49afade31b646d72bcf1 [CANDIDATE]: Term 1 election: Requested vote from peers 4914fb8984884648912625357ebdfe4b (127.0.97.2:45425), b3fbc329151a4176903c2fdc296eeaf3 (127.0.97.3:42829)
I20260929 21:04:19.606750 19019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c4fea01aa9042b6a2dcd099db5749f8" candidate_uuid: "1071939dd7de49afade31b646d72bcf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3fbc329151a4176903c2fdc296eeaf3"
I20260929 21:04:19.606746 18886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c4fea01aa9042b6a2dcd099db5749f8" candidate_uuid: "1071939dd7de49afade31b646d72bcf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4914fb8984884648912625357ebdfe4b"
I20260929 21:04:19.606962 19019 raft_consensus.cc:3037] T 5c4fea01aa9042b6a2dcd099db5749f8 P b3fbc329151a4176903c2fdc296eeaf3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:19.606962 18886 raft_consensus.cc:3037] T 5c4fea01aa9042b6a2dcd099db5749f8 P 4914fb8984884648912625357ebdfe4b [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:19.610211 19019 raft_consensus.cc:2454] T 5c4fea01aa9042b6a2dcd099db5749f8 P b3fbc329151a4176903c2fdc296eeaf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1071939dd7de49afade31b646d72bcf1 in term 1.
I20260929 21:04:19.610203 18886 raft_consensus.cc:2454] T 5c4fea01aa9042b6a2dcd099db5749f8 P 4914fb8984884648912625357ebdfe4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1071939dd7de49afade31b646d72bcf1 in term 1.
I20260929 21:04:19.610594 18688 leader_election.cc:302] T 5c4fea01aa9042b6a2dcd099db5749f8 P 1071939dd7de49afade31b646d72bcf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1071939dd7de49afade31b646d72bcf1, b3fbc329151a4176903c2fdc296eeaf3; no voters:
I20260929 21:04:19.610918 19088 raft_consensus.cc:2781] T 5c4fea01aa9042b6a2dcd099db5749f8 P 1071939dd7de49afade31b646d72bcf1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:04:19.611377 19088 raft_consensus.cc:686] T 5c4fea01aa9042b6a2dcd099db5749f8 P 1071939dd7de49afade31b646d72bcf1 [term 1 LEADER]: Becoming Leader. State: Replica: 1071939dd7de49afade31b646d72bcf1, State: Running, Role: LEADER
I20260929 21:04:19.611845 19088 consensus_queue.cc:237] T 5c4fea01aa9042b6a2dcd099db5749f8 P 1071939dd7de49afade31b646d72bcf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4914fb8984884648912625357ebdfe4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45425 } } peers { permanent_uuid: "b3fbc329151a4176903c2fdc296eeaf3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42829 } } peers { permanent_uuid: "1071939dd7de49afade31b646d72bcf1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36879 } }
I20260929 21:04:19.616192 18607 catalog_manager.cc:5495] T 5c4fea01aa9042b6a2dcd099db5749f8 P 1071939dd7de49afade31b646d72bcf1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1071939dd7de49afade31b646d72bcf1 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "1071939dd7de49afade31b646d72bcf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4914fb8984884648912625357ebdfe4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3fbc329151a4176903c2fdc296eeaf3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1071939dd7de49afade31b646d72bcf1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36879 } health_report { overall_health: HEALTHY } } }
I20260929 21:04:19.641554 19081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:19.636321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52540 (local address 127.0.97.1:36879)
0929 21:04:19.636514 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:19.636518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:19.636621 (+ 103us) server_negotiation.cc:407] Connection header received
0929 21:04:19.636794 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:19.636798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:19.636947 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:19.637077 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:04:19.637439 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:19.638132 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:19.639498 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:19.639891 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:19.639986 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:19.640439 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:04:19.641096 (+ 657us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:04:19.641226 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:19.641320 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 21:04:19.641358 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 21:04:19.649464 19019 raft_consensus.cc:1261] T 5c4fea01aa9042b6a2dcd099db5749f8 P b3fbc329151a4176903c2fdc296eeaf3 [term 1 FOLLOWER]: Refusing update from remote peer 1071939dd7de49afade31b646d72bcf1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:04:19.649463 18886 raft_consensus.cc:1261] T 5c4fea01aa9042b6a2dcd099db5749f8 P 4914fb8984884648912625357ebdfe4b [term 1 FOLLOWER]: Refusing update from remote peer 1071939dd7de49afade31b646d72bcf1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:04:19.650458 19093 consensus_queue.cc:1035] T 5c4fea01aa9042b6a2dcd099db5749f8 P 1071939dd7de49afade31b646d72bcf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4914fb8984884648912625357ebdfe4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:04:19.650842 19088 consensus_queue.cc:1035] T 5c4fea01aa9042b6a2dcd099db5749f8 P 1071939dd7de49afade31b646d72bcf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3fbc329151a4176903c2fdc296eeaf3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:04:19.662601 19096 mvcc.cc:205] Tried to move back new op lower bound from 7334772161114279936 to 7334772160974020608. Current Snapshot: MvccSnapshot[applied={T|T < 7334772161114279936}]
I20260929 21:04:19.664479 19097 mvcc.cc:205] Tried to move back new op lower bound from 7334772161114279936 to 7334772160974020608. Current Snapshot: MvccSnapshot[applied={T|T < 7334772161114279936}]
I20260929 21:04:19.676280 18606 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 21:04:19.676568 18606 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 21:04:19.680171 18606 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a110ee0adb634e6fab484e2f0fc9024b: Sending DeleteTablet for 3 replicas of tablet 5c4fea01aa9042b6a2dcd099db5749f8
I20260929 21:04:19.681524 18866 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c4fea01aa9042b6a2dcd099db5749f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:04:19 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37698
I20260929 21:04:19.681593 18999 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c4fea01aa9042b6a2dcd099db5749f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:04:19 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48348
I20260929 21:04:19.681876 18732 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c4fea01aa9042b6a2dcd099db5749f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:04:19 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52538
I20260929 21:04:19.687603 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 18669
I20260929 21:04:19.695968 19108 tablet_replica.cc:330] T 5c4fea01aa9042b6a2dcd099db5749f8 P 1071939dd7de49afade31b646d72bcf1: stopping tablet replica
I20260929 21:04:19.696713 19108 raft_consensus.cc:2229] T 5c4fea01aa9042b6a2dcd099db5749f8 P 1071939dd7de49afade31b646d72bcf1 [term 1 LEADER]: Raft consensus shutting down.
I20260929 21:04:19.697691 19108 raft_consensus.cc:2258] T 5c4fea01aa9042b6a2dcd099db5749f8 P 1071939dd7de49afade31b646d72bcf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:04:19.700647 19108 ts_tablet_manager.cc:1905] T 5c4fea01aa9042b6a2dcd099db5749f8 P 1071939dd7de49afade31b646d72bcf1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:04:19.701314 19107 tablet_replica.cc:330] T 5c4fea01aa9042b6a2dcd099db5749f8 P b3fbc329151a4176903c2fdc296eeaf3: stopping tablet replica
I20260929 21:04:19.701737 19107 raft_consensus.cc:2229] T 5c4fea01aa9042b6a2dcd099db5749f8 P b3fbc329151a4176903c2fdc296eeaf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:04:19.702232 19107 raft_consensus.cc:2258] T 5c4fea01aa9042b6a2dcd099db5749f8 P b3fbc329151a4176903c2fdc296eeaf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:04:19.709228 19108 ts_tablet_manager.cc:1918] T 5c4fea01aa9042b6a2dcd099db5749f8 P 1071939dd7de49afade31b646d72bcf1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:04:19.709425 19108 log.cc:1196] T 5c4fea01aa9042b6a2dcd099db5749f8 P 1071939dd7de49afade31b646d72bcf1: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-0/wal/wals/5c4fea01aa9042b6a2dcd099db5749f8
I20260929 21:04:19.709952 19108 ts_tablet_manager.cc:1939] T 5c4fea01aa9042b6a2dcd099db5749f8 P 1071939dd7de49afade31b646d72bcf1: Deleting consensus metadata
I20260929 21:04:19.710916 19106 tablet_replica.cc:330] T 5c4fea01aa9042b6a2dcd099db5749f8 P 4914fb8984884648912625357ebdfe4b: stopping tablet replica
I20260929 21:04:19.711371 19106 raft_consensus.cc:2229] T 5c4fea01aa9042b6a2dcd099db5749f8 P 4914fb8984884648912625357ebdfe4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:04:19.711860 19106 raft_consensus.cc:2258] T 5c4fea01aa9042b6a2dcd099db5749f8 P 4914fb8984884648912625357ebdfe4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:04:19.711898 18594 catalog_manager.cc:4841] TS 1071939dd7de49afade31b646d72bcf1 (127.0.97.1:36879): tablet 5c4fea01aa9042b6a2dcd099db5749f8 (table test-table [id=483cd24e677e483b8a6e1659a1156465]) successfully deleted
I20260929 21:04:19.714131 19106 ts_tablet_manager.cc:1905] T 5c4fea01aa9042b6a2dcd099db5749f8 P 4914fb8984884648912625357ebdfe4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:04:19.714321 19107 ts_tablet_manager.cc:1905] T 5c4fea01aa9042b6a2dcd099db5749f8 P b3fbc329151a4176903c2fdc296eeaf3: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 21:04:19 dist-test-slave-1dms krb5kdc[18562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 21:04:19.721123 19106 ts_tablet_manager.cc:1918] T 5c4fea01aa9042b6a2dcd099db5749f8 P 4914fb8984884648912625357ebdfe4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:04:19.721282 19106 log.cc:1196] T 5c4fea01aa9042b6a2dcd099db5749f8 P 4914fb8984884648912625357ebdfe4b: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-1/wal/wals/5c4fea01aa9042b6a2dcd099db5749f8
I20260929 21:04:19.721792 19106 ts_tablet_manager.cc:1939] T 5c4fea01aa9042b6a2dcd099db5749f8 P 4914fb8984884648912625357ebdfe4b: Deleting consensus metadata
I20260929 21:04:19.722070 19107 ts_tablet_manager.cc:1918] T 5c4fea01aa9042b6a2dcd099db5749f8 P b3fbc329151a4176903c2fdc296eeaf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:04:19.722527 19107 log.cc:1196] T 5c4fea01aa9042b6a2dcd099db5749f8 P b3fbc329151a4176903c2fdc296eeaf3: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790715556572573-388-0/minicluster-data/ts-2/wal/wals/5c4fea01aa9042b6a2dcd099db5749f8
I20260929 21:04:19.723064 19107 ts_tablet_manager.cc:1939] T 5c4fea01aa9042b6a2dcd099db5749f8 P b3fbc329151a4176903c2fdc296eeaf3: Deleting consensus metadata
I20260929 21:04:19.723657 18592 catalog_manager.cc:4841] TS 4914fb8984884648912625357ebdfe4b (127.0.97.2:45425): tablet 5c4fea01aa9042b6a2dcd099db5749f8 (table test-table [id=483cd24e677e483b8a6e1659a1156465]) successfully deleted
I20260929 21:04:19.724673 18594 catalog_manager.cc:4841] TS b3fbc329151a4176903c2fdc296eeaf3 (127.0.97.3:42829): tablet 5c4fea01aa9042b6a2dcd099db5749f8 (table test-table [id=483cd24e677e483b8a6e1659a1156465]) successfully deleted
I20260929 21:04:19.727772 19081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:19.688363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52546 (local address 127.0.97.1:36879)
0929 21:04:19.688562 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:19.688568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:19.707665 (+ 19097us) server_negotiation.cc:407] Connection header received
0929 21:04:19.707877 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:19.707889 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:19.707986 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:19.708096 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:19.710316 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:19.711547 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:19.713683 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:19.714360 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:19.714461 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:19.720137 (+ 5676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:19.720154 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:19.720172 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:04:19.720221 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:19.724371 (+ 4150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:19.725281 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:19.725288 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:19.725290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:19.725428 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:19.725836 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:19.725839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:19.725841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:19.726170 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:19.726431 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:19.727002 (+ 571us) server_negotiation.cc:299] Negotiation successful
0929 21:04:19.727431 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 21:04:19.728470 18792 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:04:19.827919 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 18669
I20260929 21:04:19.846068 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 18803
Sep 29 21:04:19 dist-test-slave-1dms krb5kdc[18562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 21:04:19.860584 19078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:19.846924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37714 (local address 127.0.97.2:45425)
0929 21:04:19.847111 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:19.847117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:19.847138 (+ 21us) server_negotiation.cc:407] Connection header received
0929 21:04:19.847316 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:19.847322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:19.847415 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:19.847531 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:19.848883 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:19.849620 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:19.850880 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:19.851264 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:19.851377 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:19.855523 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:19.855536 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:19.855552 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:04:19.855584 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:19.858263 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:19.858886 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:19.858893 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:19.858895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:19.859006 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:19.859304 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:19.859307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:19.859308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:19.859674 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:19.859877 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:19.860127 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 21:04:19.860285 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 21:04:19.861322 18926 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:04:19.949162 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 18803
I20260929 21:04:19.966527 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 18936
Sep 29 21:04:19 dist-test-slave-1dms krb5kdc[18562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 21:04:19.981611 19079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:19.967340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48364 (local address 127.0.97.3:42829)
0929 21:04:19.967546 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:19.967551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:19.967594 (+ 43us) server_negotiation.cc:407] Connection header received
0929 21:04:19.967659 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:19.967663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:19.967729 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:19.967814 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:19.969174 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:19.969948 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:19.971171 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:19.971556 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:19.971646 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:19.975977 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:19.975993 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:19.976010 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:04:19.976051 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:19.979015 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:19.979789 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:19.979796 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:19.979799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:19.979946 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:19.980274 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:19.980277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:19.980279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:19.980642 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:19.980990 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:19.981192 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 21:04:19.981362 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 21:04:19.982259 19059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:04:20.069224 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 18936
I20260929 21:04:20.085822 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 18575
I20260929 21:04:20.086906 18637 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:04:20.197021 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 18575
2026-09-29T21:04:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4116 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T21:04:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:04:20Z Disabled control of system clock
I20260929 21:04:20.272053 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42963
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 21:04:20.601418 19117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:20.601697 19117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:20.601778 19117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:20.609488 19117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:04:20.609635 19117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:20.609685 19117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:20.609741 19117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:04:20.609783 19117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:04:20.618829 19117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42963
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:20.620050 19117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:20.621838 19117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:20.633694 19123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:20.633664 19125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:20.633749 19122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:20.634697 19117 server_base.cc:1034] running on GCE node
I20260929 21:04:20.635519 19117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:20.637045 19117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:20.638298 19117 hybrid_clock.cc:648] HybridClock initialized: now 1790715860638232 us; error 64 us; skew 500 ppm
I20260929 21:04:20.638687 19117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 21:04:20.641111 19117 webserver.cc:460] Webserver started at http://127.0.97.62:37831/ using document root <none> and password file <none>
I20260929 21:04:20.641821 19117 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:20.641957 19117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:20.642290 19117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:20.645323 19117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "ba7d6b0e3d454554969629e9fc3c967b"
format_stamp: "Formatted at 2026-09-29 21:04:20 on dist-test-slave-1dms"
server_key: "63d4bcaa7489953be9a6c92e4fe9eac9"
server_key_iv: "bf15f7ea9cb3e10b9b1a03ec62cd015b"
server_key_version: "encryptionkey@0"
I20260929 21:04:20.646164 19117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "ba7d6b0e3d454554969629e9fc3c967b"
format_stamp: "Formatted at 2026-09-29 21:04:20 on dist-test-slave-1dms"
server_key: "63d4bcaa7489953be9a6c92e4fe9eac9"
server_key_iv: "bf15f7ea9cb3e10b9b1a03ec62cd015b"
server_key_version: "encryptionkey@0"
I20260929 21:04:20.651173 19117 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260929 21:04:20.654539 19131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:20.655854 19117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 21:04:20.656034 19117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "ba7d6b0e3d454554969629e9fc3c967b"
format_stamp: "Formatted at 2026-09-29 21:04:20 on dist-test-slave-1dms"
server_key: "63d4bcaa7489953be9a6c92e4fe9eac9"
server_key_iv: "bf15f7ea9cb3e10b9b1a03ec62cd015b"
server_key_version: "encryptionkey@0"
I20260929 21:04:20.656251 19117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:20.667631 19117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:20.668452 19117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:20.668736 19117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:20.687062 19117 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42963
I20260929 21:04:20.687081 19182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42963 every 8 connection(s)
I20260929 21:04:20.688380 19117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:04:20.692407 19183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:04:20.693439 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 19117
I20260929 21:04:20.693624 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:04:20.694232 388 external_mini_cluster.cc:1428] Setting key 49fe96805ea3bf11c38ce30465c3c0e3
I20260929 21:04:20.706318 19183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba7d6b0e3d454554969629e9fc3c967b: Bootstrap starting.
I20260929 21:04:20.708642 19186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:20.696364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52886 (local address 127.0.97.62:42963)
0929 21:04:20.697669 (+ 1305us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:20.697692 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:20.697750 (+ 58us) server_negotiation.cc:407] Connection header received
0929 21:04:20.700033 (+ 2283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:20.700063 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:20.700654 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:20.701076 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:20.701624 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:20.702858 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:20.704263 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:20.705022 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:20.705220 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:20.706388 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:20.706395 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:20.706407 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:20.706982 (+ 575us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:20.707058 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:20.707078 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:20.707204 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:20.707408 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 21:04:20.707722 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":769,"thread_start_us":494,"threads_started":1}
I20260929 21:04:20.709728 19183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba7d6b0e3d454554969629e9fc3c967b: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:20.710763 19183 log.cc:824] T 00000000000000000000000000000000 P ba7d6b0e3d454554969629e9fc3c967b: Log is configured to *not* fsync() on all Append() calls
I20260929 21:04:20.713438 19183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba7d6b0e3d454554969629e9fc3c967b: No bootstrap required, opened a new log
I20260929 21:04:20.720228 19183 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba7d6b0e3d454554969629e9fc3c967b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7d6b0e3d454554969629e9fc3c967b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42963 } }
I20260929 21:04:20.720597 19183 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba7d6b0e3d454554969629e9fc3c967b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:20.720719 19183 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba7d6b0e3d454554969629e9fc3c967b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba7d6b0e3d454554969629e9fc3c967b, State: Initialized, Role: FOLLOWER
I20260929 21:04:20.721340 19183 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba7d6b0e3d454554969629e9fc3c967b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7d6b0e3d454554969629e9fc3c967b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42963 } }
I20260929 21:04:20.721575 19183 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba7d6b0e3d454554969629e9fc3c967b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:04:20.721714 19183 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba7d6b0e3d454554969629e9fc3c967b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:04:20.721886 19183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba7d6b0e3d454554969629e9fc3c967b [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:20.723918 19183 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba7d6b0e3d454554969629e9fc3c967b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7d6b0e3d454554969629e9fc3c967b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42963 } }
I20260929 21:04:20.724453 19183 leader_election.cc:302] T 00000000000000000000000000000000 P ba7d6b0e3d454554969629e9fc3c967b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba7d6b0e3d454554969629e9fc3c967b; no voters:
I20260929 21:04:20.725081 19183 leader_election.cc:288] T 00000000000000000000000000000000 P ba7d6b0e3d454554969629e9fc3c967b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 21:04:20.725207 19188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba7d6b0e3d454554969629e9fc3c967b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:04:20.725934 19188 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba7d6b0e3d454554969629e9fc3c967b [term 1 LEADER]: Becoming Leader. State: Replica: ba7d6b0e3d454554969629e9fc3c967b, State: Running, Role: LEADER
I20260929 21:04:20.726485 19188 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba7d6b0e3d454554969629e9fc3c967b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7d6b0e3d454554969629e9fc3c967b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42963 } }
I20260929 21:04:20.727169 19183 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba7d6b0e3d454554969629e9fc3c967b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:04:20.729261 19190 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba7d6b0e3d454554969629e9fc3c967b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba7d6b0e3d454554969629e9fc3c967b. Latest consensus state: current_term: 1 leader_uuid: "ba7d6b0e3d454554969629e9fc3c967b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7d6b0e3d454554969629e9fc3c967b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42963 } } }
I20260929 21:04:20.729204 19189 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba7d6b0e3d454554969629e9fc3c967b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba7d6b0e3d454554969629e9fc3c967b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7d6b0e3d454554969629e9fc3c967b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42963 } } }
I20260929 21:04:20.729664 19189 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba7d6b0e3d454554969629e9fc3c967b [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:20.729668 19190 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba7d6b0e3d454554969629e9fc3c967b [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:20.731812 19195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:04:20.736310 19195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:04:20.744454 19195 catalog_manager.cc:1347] Generated new cluster ID: 1bb6de1c7bc647a9a622173edda777a7
I20260929 21:04:20.744568 19195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:04:20.755827 19195 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:04:20.757162 19195 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:04:20.768427 19195 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba7d6b0e3d454554969629e9fc3c967b: Generated new TSK 0
I20260929 21:04:20.769366 19195 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 21:04:20.777371 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42963
--builtin_ntp_servers=127.0.97.20:39541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 21:04:21.107770 19207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:21.108052 19207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:21.108165 19207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:21.115429 19207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:21.115526 19207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:21.115666 19207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 21:04:21.124739 19207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:21.126050 19207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:21.128008 19207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:21.141242 19212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:21.141332 19215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:21.141188 19213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:21.142342 19207 server_base.cc:1034] running on GCE node
I20260929 21:04:21.142951 19207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:21.143862 19207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:21.145082 19207 hybrid_clock.cc:648] HybridClock initialized: now 1790715861145039 us; error 54 us; skew 500 ppm
I20260929 21:04:21.145397 19207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 21:04:21.147539 19207 webserver.cc:460] Webserver started at http://127.0.97.1:35275/ using document root <none> and password file <none>
I20260929 21:04:21.148243 19207 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:21.148375 19207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:21.148699 19207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:21.151546 19207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-0/data/instance:
uuid: "0ec20e77fc4546aab34c3db7908cc971"
format_stamp: "Formatted at 2026-09-29 21:04:21 on dist-test-slave-1dms"
server_key: "c5a938e1b3cfd2f66c7b6753a5ba6aa8"
server_key_iv: "8d7d2a4c91697ab1cfd6d782bb5aebed"
server_key_version: "encryptionkey@0"
I20260929 21:04:21.152341 19207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0ec20e77fc4546aab34c3db7908cc971"
format_stamp: "Formatted at 2026-09-29 21:04:21 on dist-test-slave-1dms"
server_key: "c5a938e1b3cfd2f66c7b6753a5ba6aa8"
server_key_iv: "8d7d2a4c91697ab1cfd6d782bb5aebed"
server_key_version: "encryptionkey@0"
I20260929 21:04:21.157217 19207 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 21:04:21.160579 19221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:21.161888 19207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:04:21.162065 19207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-0/wal
uuid: "0ec20e77fc4546aab34c3db7908cc971"
format_stamp: "Formatted at 2026-09-29 21:04:21 on dist-test-slave-1dms"
server_key: "c5a938e1b3cfd2f66c7b6753a5ba6aa8"
server_key_iv: "8d7d2a4c91697ab1cfd6d782bb5aebed"
server_key_version: "encryptionkey@0"
I20260929 21:04:21.162278 19207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:21.189378 19207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:21.190235 19207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:21.190516 19207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:21.191460 19207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:04:21.192963 19207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:04:21.193078 19207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:21.193214 19207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:04:21.193302 19207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:21.225818 19207 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35209
I20260929 21:04:21.225903 19333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35209 every 8 connection(s)
I20260929 21:04:21.227114 19207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
I20260929 21:04:21.231591 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 19207
I20260929 21:04:21.231765 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-0/wal/instance
I20260929 21:04:21.232227 388 external_mini_cluster.cc:1428] Setting key ef8312cb99e5f8dc46514d798f904082
I20260929 21:04:21.234840 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42963
--builtin_ntp_servers=127.0.97.20:39541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 21:04:21.239961 19336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:21.229736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50489 (local address 127.0.97.62:42963)
0929 21:04:21.230174 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:21.230178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:21.231287 (+ 1109us) server_negotiation.cc:407] Connection header received
0929 21:04:21.233475 (+ 2188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:21.233481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:21.233568 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:21.233685 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:21.234745 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:21.235529 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:21.237145 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:21.237548 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:21.237687 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:21.239082 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:21.239087 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:21.239091 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:21.239108 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:21.239150 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:21.239162 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:21.239260 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:21.239500 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 21:04:21.239597 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":246,"threads_started":1}
I20260929 21:04:21.241091 19337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:21.230162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42963 (local address 127.0.97.1:50489)
0929 21:04:21.231125 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:21.231168 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:21.233222 (+ 2054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:21.233924 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:21.233932 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:21.234021 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:04:21.234571 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:21.234591 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:21.235695 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:21.235700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:21.236985 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:21.236997 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:21.237803 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:21.237812 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:21.238055 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:21.238756 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 21:04:21.238758 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:04:21.238802 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 21:04:21.238877 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:04:21.239259 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:04:21.239271 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:04:21.239282 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 21:04:21.239478 (+ 196us) client_negotiation.cc:240] Negotiation successful
0929 21:04:21.239934 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":250,"threads_started":1}
I20260929 21:04:21.242650 19334 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42963
I20260929 21:04:21.242966 19334 heartbeater.cc:464] Registering TS with master...
I20260929 21:04:21.243709 19334 heartbeater.cc:511] Master 127.0.97.62:42963 requested a full tablet report, sending...
I20260929 21:04:21.245702 19148 ts_manager.cc:194] Registered new tserver with Master: 0ec20e77fc4546aab34c3db7908cc971 (127.0.97.1:35209)
I20260929 21:04:21.247174 19148 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:50489
W20260929 21:04:21.557051 19338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:21.557358 19338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:21.557518 19338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:21.564607 19338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:21.564749 19338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:21.564952 19338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 21:04:21.574025 19338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:21.575322 19338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:21.577097 19338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:21.589819 19344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:21.589854 19343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:21.589902 19346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:21.590922 19338 server_base.cc:1034] running on GCE node
I20260929 21:04:21.591538 19338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:21.592444 19338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:21.593660 19338 hybrid_clock.cc:648] HybridClock initialized: now 1790715861593610 us; error 44 us; skew 500 ppm
I20260929 21:04:21.593971 19338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 21:04:21.596114 19338 webserver.cc:460] Webserver started at http://127.0.97.2:45717/ using document root <none> and password file <none>
I20260929 21:04:21.596771 19338 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:21.596915 19338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:21.597226 19338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:21.600009 19338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-1/data/instance:
uuid: "f3a1061676b14a77b7206c9ab11d7225"
format_stamp: "Formatted at 2026-09-29 21:04:21 on dist-test-slave-1dms"
server_key: "6896b74107a9a7bb190c6efd38ada9b9"
server_key_iv: "96e8e9cfd16795416dd83672e7abcbe5"
server_key_version: "encryptionkey@0"
I20260929 21:04:21.600761 19338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f3a1061676b14a77b7206c9ab11d7225"
format_stamp: "Formatted at 2026-09-29 21:04:21 on dist-test-slave-1dms"
server_key: "6896b74107a9a7bb190c6efd38ada9b9"
server_key_iv: "96e8e9cfd16795416dd83672e7abcbe5"
server_key_version: "encryptionkey@0"
I20260929 21:04:21.605593 19338 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 21:04:21.608938 19352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:21.610265 19338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 21:04:21.610426 19338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-1/wal
uuid: "f3a1061676b14a77b7206c9ab11d7225"
format_stamp: "Formatted at 2026-09-29 21:04:21 on dist-test-slave-1dms"
server_key: "6896b74107a9a7bb190c6efd38ada9b9"
server_key_iv: "96e8e9cfd16795416dd83672e7abcbe5"
server_key_version: "encryptionkey@0"
I20260929 21:04:21.610631 19338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:21.625193 19338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:21.625973 19338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:21.626247 19338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:21.627241 19338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:04:21.628738 19338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:04:21.628872 19338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:21.629007 19338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:04:21.629099 19338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:21.660354 19338 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43923
I20260929 21:04:21.660356 19464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43923 every 8 connection(s)
I20260929 21:04:21.661888 19338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
I20260929 21:04:21.670753 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 19338
I20260929 21:04:21.670917 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-1/wal/instance
I20260929 21:04:21.671482 388 external_mini_cluster.cc:1428] Setting key 42bc9d6b2d838d91332644d712878393
I20260929 21:04:21.674206 19336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:21.664361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42857 (local address 127.0.97.62:42963)
0929 21:04:21.664540 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:21.664545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:21.666063 (+ 1518us) server_negotiation.cc:407] Connection header received
0929 21:04:21.667818 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:21.667822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:21.667889 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:21.667979 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:21.669007 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:21.669718 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:21.671286 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:21.671893 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:21.672029 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:21.673430 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:21.673436 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:21.673439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:21.673455 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:21.673485 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:21.673496 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:21.673588 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:21.673813 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 21:04:21.673907 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 21:04:21.674510 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42963
--builtin_ntp_servers=127.0.97.20:39541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 21:04:21.675200 19467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:21.664991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42963 (local address 127.0.97.2:42857)
0929 21:04:21.665912 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:21.665948 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:21.667580 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:21.668181 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:21.668189 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:21.668274 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:04:21.668811 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:21.668826 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:21.669849 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:21.669853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:21.671148 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:21.671160 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:21.672121 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:21.672128 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:21.672402 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:21.673107 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 21:04:21.673110 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:04:21.673146 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 21:04:21.673223 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:04:21.673600 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:04:21.673612 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:04:21.673625 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 21:04:21.673787 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 21:04:21.674145 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":252,"threads_started":1}
I20260929 21:04:21.676542 19465 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42963
I20260929 21:04:21.676826 19465 heartbeater.cc:464] Registering TS with master...
I20260929 21:04:21.677541 19465 heartbeater.cc:511] Master 127.0.97.62:42963 requested a full tablet report, sending...
I20260929 21:04:21.678923 19148 ts_manager.cc:194] Registered new tserver with Master: f3a1061676b14a77b7206c9ab11d7225 (127.0.97.2:43923)
I20260929 21:04:21.680156 19148 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:42857
W20260929 21:04:21.993729 19468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:21.994020 19468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:21.994164 19468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:22.001191 19468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:22.001302 19468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:22.001459 19468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 21:04:22.010201 19468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:22.011431 19468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:22.013187 19468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:22.029096 19473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:22.029058 19474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:22.029909 19468 server_base.cc:1034] running on GCE node
W20260929 21:04:22.029150 19476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:22.033649 19468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:22.034574 19468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:22.035791 19468 hybrid_clock.cc:648] HybridClock initialized: now 1790715862035758 us; error 30 us; skew 500 ppm
I20260929 21:04:22.036088 19468 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 21:04:22.038462 19468 webserver.cc:460] Webserver started at http://127.0.97.3:40489/ using document root <none> and password file <none>
I20260929 21:04:22.039106 19468 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:22.039220 19468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:22.039529 19468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:22.042297 19468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-2/data/instance:
uuid: "13cccb8ae715406bbe1722233cf77b38"
format_stamp: "Formatted at 2026-09-29 21:04:22 on dist-test-slave-1dms"
server_key: "1591d1b7534db5467df408ad4fd88b3e"
server_key_iv: "545295b4d3d566a796e312efafe7e1a7"
server_key_version: "encryptionkey@0"
I20260929 21:04:22.043035 19468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-2/wal/instance:
uuid: "13cccb8ae715406bbe1722233cf77b38"
format_stamp: "Formatted at 2026-09-29 21:04:22 on dist-test-slave-1dms"
server_key: "1591d1b7534db5467df408ad4fd88b3e"
server_key_iv: "545295b4d3d566a796e312efafe7e1a7"
server_key_version: "encryptionkey@0"
I20260929 21:04:22.047966 19468 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 21:04:22.051431 19482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:22.052909 19468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:04:22.053066 19468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-2/wal
uuid: "13cccb8ae715406bbe1722233cf77b38"
format_stamp: "Formatted at 2026-09-29 21:04:22 on dist-test-slave-1dms"
server_key: "1591d1b7534db5467df408ad4fd88b3e"
server_key_iv: "545295b4d3d566a796e312efafe7e1a7"
server_key_version: "encryptionkey@0"
I20260929 21:04:22.053279 19468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:22.068274 19468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:22.069089 19468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:22.069350 19468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:22.070291 19468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:04:22.071732 19468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:04:22.071813 19468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:22.071960 19468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:04:22.072034 19468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:22.108505 19468 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38047
I20260929 21:04:22.108587 19594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38047 every 8 connection(s)
I20260929 21:04:22.109812 19468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
I20260929 21:04:22.118350 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 19468
I20260929 21:04:22.118531 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-2/wal/instance
I20260929 21:04:22.118959 388 external_mini_cluster.cc:1428] Setting key 3fbbfb9d79679f6c57de228765f2a114
I20260929 21:04:22.123939 19336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:22.112901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43997 (local address 127.0.97.62:42963)
0929 21:04:22.113108 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:22.113114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:22.114625 (+ 1511us) server_negotiation.cc:407] Connection header received
0929 21:04:22.116326 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:22.116335 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:22.116446 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:22.116578 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:22.117599 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:22.119002 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:22.121126 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:22.121569 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:22.121706 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:22.123119 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:22.123124 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:22.123128 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:22.123144 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:22.123180 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:22.123192 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:22.123285 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:22.123541 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 21:04:22.123634 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 21:04:22.124899 19597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:22.113445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42963 (local address 127.0.97.3:43997)
0929 21:04:22.114466 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:22.114509 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:22.116051 (+ 1542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:22.116756 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:22.116763 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:22.116896 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:04:22.117441 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:22.117457 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:22.119192 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:22.119197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:22.120969 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:22.120982 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:22.121791 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:22.121798 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:22.122085 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:22.122828 (+ 743us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 21:04:22.122830 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:04:22.122864 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 21:04:22.122930 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:04:22.123285 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:04:22.123298 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:04:22.123307 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 21:04:22.123484 (+ 177us) client_negotiation.cc:240] Negotiation successful
0929 21:04:22.123844 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":347,"threads_started":1}
I20260929 21:04:22.126200 19595 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42963
I20260929 21:04:22.126539 19595 heartbeater.cc:464] Registering TS with master...
I20260929 21:04:22.127502 19595 heartbeater.cc:511] Master 127.0.97.62:42963 requested a full tablet report, sending...
I20260929 21:04:22.129163 19148 ts_manager.cc:194] Registered new tserver with Master: 13cccb8ae715406bbe1722233cf77b38 (127.0.97.3:38047)
I20260929 21:04:22.130460 19148 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:43997
I20260929 21:04:22.134199 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 21:04:22.145874 19336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:22.138804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52902 (local address 127.0.97.62:42963)
0929 21:04:22.139035 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:22.139040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:22.139464 (+ 424us) server_negotiation.cc:407] Connection header received
0929 21:04:22.139634 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:22.139638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:22.139695 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:22.139770 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:22.140474 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:22.141218 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:22.143523 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:22.143896 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:22.143980 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:22.145186 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:22.145190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:22.145194 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:22.145209 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:22.145238 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:22.145249 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:22.145355 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:22.145529 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 21:04:22.145593 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 21:04:22.149471 19148 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 21:04:22.156111 19148 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 21:04:22.177263 19608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:22.168003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34462 (local address 127.0.97.1:35209)
0929 21:04:22.169543 (+ 1540us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:22.169551 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:22.169585 (+ 34us) server_negotiation.cc:407] Connection header received
0929 21:04:22.169712 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:22.169719 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:22.169969 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:22.170177 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:04:22.170727 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:22.171839 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:22.174745 (+ 2906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:22.176008 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:22.176157 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:22.176448 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:22.176715 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 21:04:22.176921 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1319,"thread_start_us":1176,"threads_started":1}
I20260929 21:04:22.177924 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:22.167624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35209 (local address 127.0.0.1:34462)
0929 21:04:22.168677 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:22.168727 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:22.168936 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:22.170202 (+ 1266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:22.170210 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:22.170239 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:04:22.170563 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:22.170572 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:22.172019 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:22.172026 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:22.174546 (+ 2520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:22.174561 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:22.176220 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:22.176227 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:22.176452 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:22.176522 (+ 70us) client_negotiation.cc:769] Sending connection context
0929 21:04:22.176654 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 21:04:22.176870 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"thread_start_us":633,"threads_started":1}
I20260929 21:04:22.179179 19611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:22.168319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38047 (local address 127.0.0.1:50642)
0929 21:04:22.169722 (+ 1403us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:22.169738 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:22.169832 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:22.171600 (+ 1768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:22.171605 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:22.171629 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:04:22.172100 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:22.172115 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:22.174202 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:22.174207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:22.176301 (+ 2094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:22.176326 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:22.178481 (+ 2155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:22.178487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:22.178640 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:22.178673 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 21:04:22.178748 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 21:04:22.178862 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1292,"thread_start_us":3733,"threads_started":1}
I20260929 21:04:22.179486 19610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:22.168996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50642 (local address 127.0.97.3:38047)
0929 21:04:22.170931 (+ 1935us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:22.170942 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:22.170982 (+ 40us) server_negotiation.cc:407] Connection header received
0929 21:04:22.171096 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:22.171107 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:22.171380 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:22.171608 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:04:22.172437 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:22.174058 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:22.176545 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:22.178302 (+ 1757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:22.178432 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:22.178703 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:22.178785 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 21:04:22.178935 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1727,"thread_start_us":1567,"threads_started":1}
I20260929 21:04:22.180581 19609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:22.169251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43923 (local address 127.0.0.1:40792)
0929 21:04:22.172500 (+ 3249us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:22.172522 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:22.172644 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:22.173610 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:22.173613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:22.173630 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:04:22.173905 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:22.173912 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:22.175902 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:22.175907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:22.178094 (+ 2187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:22.178109 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:22.179910 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:22.179916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:22.180076 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:22.180107 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 21:04:22.180194 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 21:04:22.180313 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3135,"thread_start_us":319,"threads_started":1}
I20260929 21:04:22.180712 19612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:22.169800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40792 (local address 127.0.97.2:43923)
0929 21:04:22.170326 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:22.170334 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:22.172776 (+ 2442us) server_negotiation.cc:407] Connection header received
0929 21:04:22.173000 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:22.173009 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:22.173206 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:22.173383 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:04:22.174063 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:22.175691 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:22.178316 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:22.179714 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:22.179883 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:22.180093 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:22.180238 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 21:04:22.180381 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":235,"threads_started":1}
I20260929 21:04:22.181655 19269 tablet_service.cc:1467] Processing CreateTablet for tablet beb888c8564b4f0c9c474fb59321d5a0 (DEFAULT_TABLE table=test-table [id=494ecab150d4443ba68530b4a85a6c5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:04:22.182966 19269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beb888c8564b4f0c9c474fb59321d5a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:04:22.183425 19400 tablet_service.cc:1467] Processing CreateTablet for tablet beb888c8564b4f0c9c474fb59321d5a0 (DEFAULT_TABLE table=test-table [id=494ecab150d4443ba68530b4a85a6c5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:04:22.184473 19400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beb888c8564b4f0c9c474fb59321d5a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:04:22.188264 19530 tablet_service.cc:1467] Processing CreateTablet for tablet beb888c8564b4f0c9c474fb59321d5a0 (DEFAULT_TABLE table=test-table [id=494ecab150d4443ba68530b4a85a6c5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:04:22.189589 19530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beb888c8564b4f0c9c474fb59321d5a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:04:22.194697 19613 tablet_bootstrap.cc:492] T beb888c8564b4f0c9c474fb59321d5a0 P 0ec20e77fc4546aab34c3db7908cc971: Bootstrap starting.
I20260929 21:04:22.198176 19613 tablet_bootstrap.cc:654] T beb888c8564b4f0c9c474fb59321d5a0 P 0ec20e77fc4546aab34c3db7908cc971: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:22.199218 19613 log.cc:824] T beb888c8564b4f0c9c474fb59321d5a0 P 0ec20e77fc4546aab34c3db7908cc971: Log is configured to *not* fsync() on all Append() calls
I20260929 21:04:22.201931 19613 tablet_bootstrap.cc:492] T beb888c8564b4f0c9c474fb59321d5a0 P 0ec20e77fc4546aab34c3db7908cc971: No bootstrap required, opened a new log
I20260929 21:04:22.202196 19613 ts_tablet_manager.cc:1397] T beb888c8564b4f0c9c474fb59321d5a0 P 0ec20e77fc4546aab34c3db7908cc971: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 21:04:22.203346 19616 tablet_bootstrap.cc:492] T beb888c8564b4f0c9c474fb59321d5a0 P 13cccb8ae715406bbe1722233cf77b38: Bootstrap starting.
I20260929 21:04:22.205045 19614 tablet_bootstrap.cc:492] T beb888c8564b4f0c9c474fb59321d5a0 P f3a1061676b14a77b7206c9ab11d7225: Bootstrap starting.
I20260929 21:04:22.206131 19616 tablet_bootstrap.cc:654] T beb888c8564b4f0c9c474fb59321d5a0 P 13cccb8ae715406bbe1722233cf77b38: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:22.207060 19616 log.cc:824] T beb888c8564b4f0c9c474fb59321d5a0 P 13cccb8ae715406bbe1722233cf77b38: Log is configured to *not* fsync() on all Append() calls
I20260929 21:04:22.208403 19614 tablet_bootstrap.cc:654] T beb888c8564b4f0c9c474fb59321d5a0 P f3a1061676b14a77b7206c9ab11d7225: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:22.209581 19616 tablet_bootstrap.cc:492] T beb888c8564b4f0c9c474fb59321d5a0 P 13cccb8ae715406bbe1722233cf77b38: No bootstrap required, opened a new log
I20260929 21:04:22.209544 19614 log.cc:824] T beb888c8564b4f0c9c474fb59321d5a0 P f3a1061676b14a77b7206c9ab11d7225: Log is configured to *not* fsync() on all Append() calls
I20260929 21:04:22.209841 19616 ts_tablet_manager.cc:1397] T beb888c8564b4f0c9c474fb59321d5a0 P 13cccb8ae715406bbe1722233cf77b38: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 21:04:22.211858 19613 raft_consensus.cc:348] T beb888c8564b4f0c9c474fb59321d5a0 P 0ec20e77fc4546aab34c3db7908cc971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec20e77fc4546aab34c3db7908cc971" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35209 } } peers { permanent_uuid: "f3a1061676b14a77b7206c9ab11d7225" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43923 } } peers { permanent_uuid: "13cccb8ae715406bbe1722233cf77b38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38047 } }
I20260929 21:04:22.212218 19613 raft_consensus.cc:374] T beb888c8564b4f0c9c474fb59321d5a0 P 0ec20e77fc4546aab34c3db7908cc971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:22.212344 19613 raft_consensus.cc:729] T beb888c8564b4f0c9c474fb59321d5a0 P 0ec20e77fc4546aab34c3db7908cc971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec20e77fc4546aab34c3db7908cc971, State: Initialized, Role: FOLLOWER
I20260929 21:04:22.212374 19614 tablet_bootstrap.cc:492] T beb888c8564b4f0c9c474fb59321d5a0 P f3a1061676b14a77b7206c9ab11d7225: No bootstrap required, opened a new log
I20260929 21:04:22.212608 19614 ts_tablet_manager.cc:1397] T beb888c8564b4f0c9c474fb59321d5a0 P f3a1061676b14a77b7206c9ab11d7225: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260929 21:04:22.212973 19613 consensus_queue.cc:260] T beb888c8564b4f0c9c474fb59321d5a0 P 0ec20e77fc4546aab34c3db7908cc971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec20e77fc4546aab34c3db7908cc971" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35209 } } peers { permanent_uuid: "f3a1061676b14a77b7206c9ab11d7225" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43923 } } peers { permanent_uuid: "13cccb8ae715406bbe1722233cf77b38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38047 } }
I20260929 21:04:22.214099 19334 heartbeater.cc:503] Master 127.0.97.62:42963 was elected leader, sending a full tablet report...
I20260929 21:04:22.214694 19613 ts_tablet_manager.cc:1428] T beb888c8564b4f0c9c474fb59321d5a0 P 0ec20e77fc4546aab34c3db7908cc971: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260929 21:04:22.219374 19616 raft_consensus.cc:348] T beb888c8564b4f0c9c474fb59321d5a0 P 13cccb8ae715406bbe1722233cf77b38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec20e77fc4546aab34c3db7908cc971" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35209 } } peers { permanent_uuid: "f3a1061676b14a77b7206c9ab11d7225" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43923 } } peers { permanent_uuid: "13cccb8ae715406bbe1722233cf77b38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38047 } }
I20260929 21:04:22.220544 19616 raft_consensus.cc:374] T beb888c8564b4f0c9c474fb59321d5a0 P 13cccb8ae715406bbe1722233cf77b38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:22.220669 19616 raft_consensus.cc:729] T beb888c8564b4f0c9c474fb59321d5a0 P 13cccb8ae715406bbe1722233cf77b38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13cccb8ae715406bbe1722233cf77b38, State: Initialized, Role: FOLLOWER
I20260929 21:04:22.221267 19616 consensus_queue.cc:260] T beb888c8564b4f0c9c474fb59321d5a0 P 13cccb8ae715406bbe1722233cf77b38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec20e77fc4546aab34c3db7908cc971" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35209 } } peers { permanent_uuid: "f3a1061676b14a77b7206c9ab11d7225" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43923 } } peers { permanent_uuid: "13cccb8ae715406bbe1722233cf77b38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38047 } }
I20260929 21:04:22.221913 19614 raft_consensus.cc:348] T beb888c8564b4f0c9c474fb59321d5a0 P f3a1061676b14a77b7206c9ab11d7225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec20e77fc4546aab34c3db7908cc971" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35209 } } peers { permanent_uuid: "f3a1061676b14a77b7206c9ab11d7225" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43923 } } peers { permanent_uuid: "13cccb8ae715406bbe1722233cf77b38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38047 } }
I20260929 21:04:22.222289 19614 raft_consensus.cc:374] T beb888c8564b4f0c9c474fb59321d5a0 P f3a1061676b14a77b7206c9ab11d7225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:22.222301 19595 heartbeater.cc:503] Master 127.0.97.62:42963 was elected leader, sending a full tablet report...
I20260929 21:04:22.222400 19614 raft_consensus.cc:729] T beb888c8564b4f0c9c474fb59321d5a0 P f3a1061676b14a77b7206c9ab11d7225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3a1061676b14a77b7206c9ab11d7225, State: Initialized, Role: FOLLOWER
I20260929 21:04:22.222954 19616 ts_tablet_manager.cc:1428] T beb888c8564b4f0c9c474fb59321d5a0 P 13cccb8ae715406bbe1722233cf77b38: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260929 21:04:22.222934 19614 consensus_queue.cc:260] T beb888c8564b4f0c9c474fb59321d5a0 P f3a1061676b14a77b7206c9ab11d7225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec20e77fc4546aab34c3db7908cc971" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35209 } } peers { permanent_uuid: "f3a1061676b14a77b7206c9ab11d7225" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43923 } } peers { permanent_uuid: "13cccb8ae715406bbe1722233cf77b38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38047 } }
I20260929 21:04:22.223987 19465 heartbeater.cc:503] Master 127.0.97.62:42963 was elected leader, sending a full tablet report...
I20260929 21:04:22.224558 19614 ts_tablet_manager.cc:1428] T beb888c8564b4f0c9c474fb59321d5a0 P f3a1061676b14a77b7206c9ab11d7225: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
W20260929 21:04:22.229257 19335 tablet.cc:2369] T beb888c8564b4f0c9c474fb59321d5a0 P 0ec20e77fc4546aab34c3db7908cc971: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 21:04:22.275544 19619 raft_consensus.cc:482] T beb888c8564b4f0c9c474fb59321d5a0 P 0ec20e77fc4546aab34c3db7908cc971 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 21:04:22.275849 19619 raft_consensus.cc:504] T beb888c8564b4f0c9c474fb59321d5a0 P 0ec20e77fc4546aab34c3db7908cc971 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec20e77fc4546aab34c3db7908cc971" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35209 } } peers { permanent_uuid: "f3a1061676b14a77b7206c9ab11d7225" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43923 } } peers { permanent_uuid: "13cccb8ae715406bbe1722233cf77b38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38047 } }
I20260929 21:04:22.277390 19619 leader_election.cc:288] T beb888c8564b4f0c9c474fb59321d5a0 P 0ec20e77fc4546aab34c3db7908cc971 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3a1061676b14a77b7206c9ab11d7225 (127.0.97.2:43923), 13cccb8ae715406bbe1722233cf77b38 (127.0.97.3:38047)
I20260929 21:04:22.284701 19610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:22.277804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58725 (local address 127.0.97.3:38047)
0929 21:04:22.278001 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:22.278007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:22.278811 (+ 804us) server_negotiation.cc:407] Connection header received
0929 21:04:22.279123 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:22.279131 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:22.279216 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:22.279321 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:04:22.280005 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:22.280783 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:22.282971 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:22.283759 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:22.283912 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:22.284027 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:22.284310 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 21:04:22.284448 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 21:04:22.284721 19612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:22.277802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36917 (local address 127.0.97.2:43923)
0929 21:04:22.278036 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:22.278042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:22.278981 (+ 939us) server_negotiation.cc:407] Connection header received
0929 21:04:22.279184 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:22.279188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:22.279281 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:22.279366 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:04:22.280211 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:22.280992 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:22.283008 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:22.283764 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:22.283893 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:22.283992 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:22.284351 (+ 359us) server_negotiation.cc:299] Negotiation successful
0929 21:04:22.284470 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260929 21:04:22.285101 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:22.277803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38047 (local address 127.0.97.1:58725)
0929 21:04:22.278662 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:22.278700 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:22.278869 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:22.279364 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:22.279369 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:22.279401 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:04:22.279843 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:22.279858 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:22.281005 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:22.281012 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:22.282784 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:22.282796 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:22.283964 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:22.283970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:22.284142 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:22.284168 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 21:04:22.284246 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 21:04:22.284400 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"mutex_wait_us":106,"thread_start_us":430,"threads_started":1}
I20260929 21:04:22.285174 19623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:22.278085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43923 (local address 127.0.97.1:36917)
0929 21:04:22.278869 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:22.278884 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:22.278990 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:22.279507 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:22.279512 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:22.279598 (+ 86us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:04:22.280042 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:22.280056 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:22.281133 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:22.281137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:22.282805 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:22.282817 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:22.284001 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:22.284009 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:22.284179 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:22.284201 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 21:04:22.284314 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 21:04:22.284459 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":78,"spinlock_wait_cycles":24448,"thread_start_us":443,"threads_started":1}
I20260929 21:04:22.286083 19420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb888c8564b4f0c9c474fb59321d5a0" candidate_uuid: "0ec20e77fc4546aab34c3db7908cc971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3a1061676b14a77b7206c9ab11d7225" is_pre_election: true
I20260929 21:04:22.286221 19550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb888c8564b4f0c9c474fb59321d5a0" candidate_uuid: "0ec20e77fc4546aab34c3db7908cc971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13cccb8ae715406bbe1722233cf77b38" is_pre_election: true
I20260929 21:04:22.286625 19420 raft_consensus.cc:2454] T beb888c8564b4f0c9c474fb59321d5a0 P f3a1061676b14a77b7206c9ab11d7225 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ec20e77fc4546aab34c3db7908cc971 in term 0.
I20260929 21:04:22.286731 19550 raft_consensus.cc:2454] T beb888c8564b4f0c9c474fb59321d5a0 P 13cccb8ae715406bbe1722233cf77b38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ec20e77fc4546aab34c3db7908cc971 in term 0.
I20260929 21:04:22.287310 19225 leader_election.cc:302] T beb888c8564b4f0c9c474fb59321d5a0 P 0ec20e77fc4546aab34c3db7908cc971 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec20e77fc4546aab34c3db7908cc971, f3a1061676b14a77b7206c9ab11d7225; no voters:
I20260929 21:04:22.287738 19619 raft_consensus.cc:2781] T beb888c8564b4f0c9c474fb59321d5a0 P 0ec20e77fc4546aab34c3db7908cc971 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 21:04:22.287889 19619 raft_consensus.cc:482] T beb888c8564b4f0c9c474fb59321d5a0 P 0ec20e77fc4546aab34c3db7908cc971 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 21:04:22.287989 19619 raft_consensus.cc:3037] T beb888c8564b4f0c9c474fb59321d5a0 P 0ec20e77fc4546aab34c3db7908cc971 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:22.290220 19619 raft_consensus.cc:504] T beb888c8564b4f0c9c474fb59321d5a0 P 0ec20e77fc4546aab34c3db7908cc971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec20e77fc4546aab34c3db7908cc971" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35209 } } peers { permanent_uuid: "f3a1061676b14a77b7206c9ab11d7225" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43923 } } peers { permanent_uuid: "13cccb8ae715406bbe1722233cf77b38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38047 } }
I20260929 21:04:22.290834 19619 leader_election.cc:288] T beb888c8564b4f0c9c474fb59321d5a0 P 0ec20e77fc4546aab34c3db7908cc971 [CANDIDATE]: Term 1 election: Requested vote from peers f3a1061676b14a77b7206c9ab11d7225 (127.0.97.2:43923), 13cccb8ae715406bbe1722233cf77b38 (127.0.97.3:38047)
I20260929 21:04:22.291253 19420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb888c8564b4f0c9c474fb59321d5a0" candidate_uuid: "0ec20e77fc4546aab34c3db7908cc971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3a1061676b14a77b7206c9ab11d7225"
I20260929 21:04:22.291343 19550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb888c8564b4f0c9c474fb59321d5a0" candidate_uuid: "0ec20e77fc4546aab34c3db7908cc971" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13cccb8ae715406bbe1722233cf77b38"
I20260929 21:04:22.291477 19420 raft_consensus.cc:3037] T beb888c8564b4f0c9c474fb59321d5a0 P f3a1061676b14a77b7206c9ab11d7225 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:22.291522 19550 raft_consensus.cc:3037] T beb888c8564b4f0c9c474fb59321d5a0 P 13cccb8ae715406bbe1722233cf77b38 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:22.294768 19420 raft_consensus.cc:2454] T beb888c8564b4f0c9c474fb59321d5a0 P f3a1061676b14a77b7206c9ab11d7225 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ec20e77fc4546aab34c3db7908cc971 in term 1.
I20260929 21:04:22.294763 19550 raft_consensus.cc:2454] T beb888c8564b4f0c9c474fb59321d5a0 P 13cccb8ae715406bbe1722233cf77b38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ec20e77fc4546aab34c3db7908cc971 in term 1.
I20260929 21:04:22.295136 19225 leader_election.cc:302] T beb888c8564b4f0c9c474fb59321d5a0 P 0ec20e77fc4546aab34c3db7908cc971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec20e77fc4546aab34c3db7908cc971, f3a1061676b14a77b7206c9ab11d7225; no voters:
I20260929 21:04:22.295455 19619 raft_consensus.cc:2781] T beb888c8564b4f0c9c474fb59321d5a0 P 0ec20e77fc4546aab34c3db7908cc971 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:04:22.295856 19619 raft_consensus.cc:686] T beb888c8564b4f0c9c474fb59321d5a0 P 0ec20e77fc4546aab34c3db7908cc971 [term 1 LEADER]: Becoming Leader. State: Replica: 0ec20e77fc4546aab34c3db7908cc971, State: Running, Role: LEADER
I20260929 21:04:22.296315 19619 consensus_queue.cc:237] T beb888c8564b4f0c9c474fb59321d5a0 P 0ec20e77fc4546aab34c3db7908cc971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec20e77fc4546aab34c3db7908cc971" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35209 } } peers { permanent_uuid: "f3a1061676b14a77b7206c9ab11d7225" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43923 } } peers { permanent_uuid: "13cccb8ae715406bbe1722233cf77b38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38047 } }
I20260929 21:04:22.300288 19147 catalog_manager.cc:5495] T beb888c8564b4f0c9c474fb59321d5a0 P 0ec20e77fc4546aab34c3db7908cc971 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ec20e77fc4546aab34c3db7908cc971 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "0ec20e77fc4546aab34c3db7908cc971" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec20e77fc4546aab34c3db7908cc971" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3a1061676b14a77b7206c9ab11d7225" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13cccb8ae715406bbe1722233cf77b38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38047 } health_report { overall_health: UNKNOWN } } }
I20260929 21:04:22.314900 19608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:22.309870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34470 (local address 127.0.97.1:35209)
0929 21:04:22.310066 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:22.310071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:22.310087 (+ 16us) server_negotiation.cc:407] Connection header received
0929 21:04:22.310255 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:22.310258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:22.310348 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:22.310468 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:04:22.310779 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:22.311460 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:22.312751 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:22.313221 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:22.313312 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:22.313817 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:04:22.314279 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:04:22.314399 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:22.314590 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 21:04:22.314634 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 21:04:22.322544 19550 raft_consensus.cc:1261] T beb888c8564b4f0c9c474fb59321d5a0 P 13cccb8ae715406bbe1722233cf77b38 [term 1 FOLLOWER]: Refusing update from remote peer 0ec20e77fc4546aab34c3db7908cc971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:04:22.322541 19420 raft_consensus.cc:1261] T beb888c8564b4f0c9c474fb59321d5a0 P f3a1061676b14a77b7206c9ab11d7225 [term 1 FOLLOWER]: Refusing update from remote peer 0ec20e77fc4546aab34c3db7908cc971: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:04:22.323477 19624 consensus_queue.cc:1035] T beb888c8564b4f0c9c474fb59321d5a0 P 0ec20e77fc4546aab34c3db7908cc971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13cccb8ae715406bbe1722233cf77b38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:04:22.323853 19619 consensus_queue.cc:1035] T beb888c8564b4f0c9c474fb59321d5a0 P 0ec20e77fc4546aab34c3db7908cc971 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3a1061676b14a77b7206c9ab11d7225" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:04:22.334539 19629 mvcc.cc:205] Tried to move back new op lower bound from 7334772172063494144 to 7334772171969388544. Current Snapshot: MvccSnapshot[applied={T|T < 7334772172063494144}]
I20260929 21:04:22.335216 19627 mvcc.cc:205] Tried to move back new op lower bound from 7334772172063494144 to 7334772171969388544. Current Snapshot: MvccSnapshot[applied={T|T < 7334772172063494144}]
I20260929 21:04:22.349938 19148 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 21:04:22.350237 19148 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 21:04:22.353551 19148 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ba7d6b0e3d454554969629e9fc3c967b: Sending DeleteTablet for 3 replicas of tablet beb888c8564b4f0c9c474fb59321d5a0
I20260929 21:04:22.354535 19269 tablet_service.cc:1514] Processing DeleteTablet for tablet beb888c8564b4f0c9c474fb59321d5a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:04:22 UTC) from {username='slave'} at 127.0.0.1:34462
I20260929 21:04:22.354686 19400 tablet_service.cc:1514] Processing DeleteTablet for tablet beb888c8564b4f0c9c474fb59321d5a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:04:22 UTC) from {username='slave'} at 127.0.0.1:40792
I20260929 21:04:22.354820 19530 tablet_service.cc:1514] Processing DeleteTablet for tablet beb888c8564b4f0c9c474fb59321d5a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:04:22 UTC) from {username='slave'} at 127.0.0.1:50642
I20260929 21:04:22.356877 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 19207
I20260929 21:04:22.367241 19638 tablet_replica.cc:330] T beb888c8564b4f0c9c474fb59321d5a0 P 13cccb8ae715406bbe1722233cf77b38: stopping tablet replica
I20260929 21:04:22.367924 19638 raft_consensus.cc:2229] T beb888c8564b4f0c9c474fb59321d5a0 P 13cccb8ae715406bbe1722233cf77b38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:04:22.368417 19638 raft_consensus.cc:2258] T beb888c8564b4f0c9c474fb59321d5a0 P 13cccb8ae715406bbe1722233cf77b38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:04:22.371837 19638 ts_tablet_manager.cc:1905] T beb888c8564b4f0c9c474fb59321d5a0 P 13cccb8ae715406bbe1722233cf77b38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:04:22.378440 19608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:22.358471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34478 (local address 127.0.97.1:35209)
0929 21:04:22.358688 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:22.358697 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:22.363050 (+ 4353us) server_negotiation.cc:407] Connection header received
0929 21:04:22.363239 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:22.363248 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:22.367715 (+ 4467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:22.367861 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:22.369063 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:22.370665 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:22.373388 (+ 2723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:22.374308 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:22.374431 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:22.376202 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:22.376216 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:22.376238 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:22.377265 (+ 1027us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:22.377341 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:22.377372 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:22.377489 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:22.377777 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 21:04:22.377910 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 21:04:22.380321 19329 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:04:22.381072 19637 tablet_replica.cc:330] T beb888c8564b4f0c9c474fb59321d5a0 P f3a1061676b14a77b7206c9ab11d7225: stopping tablet replica
I20260929 21:04:22.382006 19637 raft_consensus.cc:2229] T beb888c8564b4f0c9c474fb59321d5a0 P f3a1061676b14a77b7206c9ab11d7225 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:04:22.383056 19637 raft_consensus.cc:2258] T beb888c8564b4f0c9c474fb59321d5a0 P f3a1061676b14a77b7206c9ab11d7225 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:04:22.383067 19638 ts_tablet_manager.cc:1918] T beb888c8564b4f0c9c474fb59321d5a0 P 13cccb8ae715406bbe1722233cf77b38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:04:22.383263 19638 log.cc:1196] T beb888c8564b4f0c9c474fb59321d5a0 P 13cccb8ae715406bbe1722233cf77b38: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-2/wal/wals/beb888c8564b4f0c9c474fb59321d5a0
I20260929 21:04:22.383906 19638 ts_tablet_manager.cc:1939] T beb888c8564b4f0c9c474fb59321d5a0 P 13cccb8ae715406bbe1722233cf77b38: Deleting consensus metadata
I20260929 21:04:22.387372 19134 catalog_manager.cc:4841] TS 13cccb8ae715406bbe1722233cf77b38 (127.0.97.3:38047): tablet beb888c8564b4f0c9c474fb59321d5a0 (table test-table [id=494ecab150d4443ba68530b4a85a6c5d]) successfully deleted
I20260929 21:04:22.388495 19637 ts_tablet_manager.cc:1905] T beb888c8564b4f0c9c474fb59321d5a0 P f3a1061676b14a77b7206c9ab11d7225: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:04:22.394318 19637 ts_tablet_manager.cc:1918] T beb888c8564b4f0c9c474fb59321d5a0 P f3a1061676b14a77b7206c9ab11d7225: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:04:22.394482 19637 log.cc:1196] T beb888c8564b4f0c9c474fb59321d5a0 P f3a1061676b14a77b7206c9ab11d7225: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790715556572573-388-0/minicluster-data/ts-1/wal/wals/beb888c8564b4f0c9c474fb59321d5a0
I20260929 21:04:22.394936 19637 ts_tablet_manager.cc:1939] T beb888c8564b4f0c9c474fb59321d5a0 P f3a1061676b14a77b7206c9ab11d7225: Deleting consensus metadata
I20260929 21:04:22.396413 19135 catalog_manager.cc:4841] TS f3a1061676b14a77b7206c9ab11d7225 (127.0.97.2:43923): tablet beb888c8564b4f0c9c474fb59321d5a0 (table test-table [id=494ecab150d4443ba68530b4a85a6c5d]) successfully deleted
I20260929 21:04:22.484030 19636 tablet_replica.cc:330] T beb888c8564b4f0c9c474fb59321d5a0 P 0ec20e77fc4546aab34c3db7908cc971: stopping tablet replica
I20260929 21:04:22.484436 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 19207
I20260929 21:04:22.504849 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 19338
W20260929 21:04:22.513056 19135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:35209 (error 108)
I20260929 21:04:22.515384 19609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:22.514930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35209 (local address 127.0.0.1:34494)
0929 21:04:22.515135 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:22.515252 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 21:04:22.515631 19135 catalog_manager.cc:4584] TS 0ec20e77fc4546aab34c3db7908cc971 (127.0.97.1:35209): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet beb888c8564b4f0c9c474fb59321d5a0: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35209: connect: Connection refused (error 111)
I20260929 21:04:22.519701 19612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:22.505842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40804 (local address 127.0.97.2:43923)
0929 21:04:22.506010 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:22.506016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:22.512752 (+ 6736us) server_negotiation.cc:407] Connection header received
0929 21:04:22.513032 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:22.513037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:22.513183 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:22.513278 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:22.513916 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:22.514961 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:22.516499 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:22.517051 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:22.517188 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:22.518327 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:22.518336 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:22.518350 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:22.518931 (+ 581us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:22.518978 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:22.518998 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:22.519130 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:22.519268 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 21:04:22.519372 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 21:04:22.520311 19460 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:04:22.569404 19609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:22.568870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35209 (local address 127.0.0.1:34498)
0929 21:04:22.569094 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:22.569212 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 21:04:22.570447 19609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:22.570064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35209 (local address 127.0.0.1:34502)
0929 21:04:22.570234 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:22.570314 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 21:04:22.609269 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 19338
I20260929 21:04:22.625797 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 19468
I20260929 21:04:22.631817 19609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:22.631250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35209 (local address 127.0.0.1:34512)
0929 21:04:22.631493 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:22.631601 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260929 21:04:22.632721 19610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:22.626598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50650 (local address 127.0.97.3:38047)
0929 21:04:22.626795 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:22.626802 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:22.626832 (+ 30us) server_negotiation.cc:407] Connection header received
0929 21:04:22.626931 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:22.626935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:22.627017 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:22.627098 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:22.627452 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:22.628236 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:22.629576 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:22.629953 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:22.630042 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:22.631106 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:22.631120 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:22.631140 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:22.631809 (+ 669us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:22.631857 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:22.631916 (+ 59us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:22.632114 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:22.632299 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 21:04:22.632417 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 21:04:22.633019 19609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:22.632552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35209 (local address 127.0.0.1:34514)
0929 21:04:22.632749 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:22.632877 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260929 21:04:22.633396 19590 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:04:22.713687 19609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:22.713225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35209 (local address 127.0.0.1:34518)
0929 21:04:22.713433 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:22.713528 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 21:04:22.714711 19609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:22.714321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35209 (local address 127.0.0.1:34522)
0929 21:04:22.714533 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:22.714600 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 21:04:22.720301 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 19468
I20260929 21:04:22.737128 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 19117
I20260929 21:04:22.738189 19178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:04:22.846733 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 19117
2026-09-29T21:04:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2650 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 21:04:22 dist-test-slave-1dms krb5kdc[19647](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 21:04:22 dist-test-slave-1dms krb5kdc[19647](info): set up 2 sockets
Sep 29 21:04:22 dist-test-slave-1dms krb5kdc[19647](info): commencing operation
krb5kdc: starting...
W20260929 21:04:23.961040 388 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.008s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 21:04:24 dist-test-slave-1dms krb5kdc[19647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T21:04:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:04:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 21:04:24.212165 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39617
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39617
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:04:24.558063 19660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:24.558331 19660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:24.558449 19660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:24.565557 19660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:04:24.565658 19660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:24.565755 19660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:24.565846 19660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:04:24.565922 19660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:04:24.575728 19660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:24.577080 19660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:24.578408 19660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:24.590078 19666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:24.590161 19668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:24.590152 19665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:24.591289 19660 server_base.cc:1034] running on GCE node
I20260929 21:04:24.592064 19660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:24.593538 19660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:24.594753 19660 hybrid_clock.cc:648] HybridClock initialized: now 1790715864594675 us; error 58 us; skew 500 ppm
I20260929 21:04:24.595054 19660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:04:24 dist-test-slave-1dms krb5kdc[19647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:04:24.604600 19660 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 21:04:24.606303 19660 webserver.cc:460] Webserver started at http://127.0.97.62:42643/ using document root <none> and password file <none>
I20260929 21:04:24.607061 19660 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:24.607199 19660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:24.607537 19660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:24.610702 19660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "3798ac8e0aa64da58cad6ecdf93bbcd5"
format_stamp: "Formatted at 2026-09-29 21:04:24 on dist-test-slave-1dms"
server_key: "5e292d76505a7c20702a53f4f8b5e445"
server_key_iv: "58e4c8c20352f0fbb645948eb70c88d4"
server_key_version: "encryptionkey@0"
I20260929 21:04:24.611502 19660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "3798ac8e0aa64da58cad6ecdf93bbcd5"
format_stamp: "Formatted at 2026-09-29 21:04:24 on dist-test-slave-1dms"
server_key: "5e292d76505a7c20702a53f4f8b5e445"
server_key_iv: "58e4c8c20352f0fbb645948eb70c88d4"
server_key_version: "encryptionkey@0"
I20260929 21:04:24.616695 19660 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 21:04:24.620115 19675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:24.621627 19660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:04:24.621822 19660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "3798ac8e0aa64da58cad6ecdf93bbcd5"
format_stamp: "Formatted at 2026-09-29 21:04:24 on dist-test-slave-1dms"
server_key: "5e292d76505a7c20702a53f4f8b5e445"
server_key_iv: "58e4c8c20352f0fbb645948eb70c88d4"
server_key_version: "encryptionkey@0"
I20260929 21:04:24.622022 19660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:24.658372 19660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:24.662384 19660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:24.662675 19660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:24.681461 19660 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39617
I20260929 21:04:24.681527 19726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39617 every 8 connection(s)
I20260929 21:04:24.682979 19660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:04:24.685729 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 19660
I20260929 21:04:24.685940 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:04:24.686484 388 external_mini_cluster.cc:1428] Setting key 7403075c7a70560a5a0079ded29fce6f
I20260929 21:04:24.687369 19727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 21:04:24 dist-test-slave-1dms krb5kdc[19647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:04:24.702116 19727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3798ac8e0aa64da58cad6ecdf93bbcd5: Bootstrap starting.
I20260929 21:04:24.705515 19727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3798ac8e0aa64da58cad6ecdf93bbcd5: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:24.706523 19727 log.cc:824] T 00000000000000000000000000000000 P 3798ac8e0aa64da58cad6ecdf93bbcd5: Log is configured to *not* fsync() on all Append() calls
I20260929 21:04:24.709438 19730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:24.688388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36428 (local address 127.0.97.62:39617)
0929 21:04:24.689202 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:24.689214 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:24.689252 (+ 38us) server_negotiation.cc:407] Connection header received
0929 21:04:24.690864 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:24.690885 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:24.691280 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:24.691708 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:24.693486 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:24.694819 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:24.696632 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:24.697350 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:24.697475 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:24.702304 (+ 4829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:24.702315 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:24.702336 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:04:24.702375 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:24.705344 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:24.706097 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:24.706104 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:24.706110 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:24.706279 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:24.706697 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:24.706703 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:24.706707 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:24.707212 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:24.707534 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:24.707963 (+ 429us) server_negotiation.cc:299] Negotiation successful
0929 21:04:24.708351 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":374,"threads_started":1}
I20260929 21:04:24.709575 19727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3798ac8e0aa64da58cad6ecdf93bbcd5: No bootstrap required, opened a new log
I20260929 21:04:24.717289 19727 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3798ac8e0aa64da58cad6ecdf93bbcd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3798ac8e0aa64da58cad6ecdf93bbcd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39617 } }
I20260929 21:04:24.717638 19727 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3798ac8e0aa64da58cad6ecdf93bbcd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:24.717746 19727 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3798ac8e0aa64da58cad6ecdf93bbcd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3798ac8e0aa64da58cad6ecdf93bbcd5, State: Initialized, Role: FOLLOWER
I20260929 21:04:24.718254 19727 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3798ac8e0aa64da58cad6ecdf93bbcd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3798ac8e0aa64da58cad6ecdf93bbcd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39617 } }
I20260929 21:04:24.718485 19727 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3798ac8e0aa64da58cad6ecdf93bbcd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:04:24.718637 19727 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3798ac8e0aa64da58cad6ecdf93bbcd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:04:24.718808 19727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3798ac8e0aa64da58cad6ecdf93bbcd5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:24.721177 19727 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3798ac8e0aa64da58cad6ecdf93bbcd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3798ac8e0aa64da58cad6ecdf93bbcd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39617 } }
I20260929 21:04:24.721715 19727 leader_election.cc:302] T 00000000000000000000000000000000 P 3798ac8e0aa64da58cad6ecdf93bbcd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3798ac8e0aa64da58cad6ecdf93bbcd5; no voters:
I20260929 21:04:24.722275 19727 leader_election.cc:288] T 00000000000000000000000000000000 P 3798ac8e0aa64da58cad6ecdf93bbcd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 21:04:24.722471 19732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3798ac8e0aa64da58cad6ecdf93bbcd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:04:24.723242 19732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3798ac8e0aa64da58cad6ecdf93bbcd5 [term 1 LEADER]: Becoming Leader. State: Replica: 3798ac8e0aa64da58cad6ecdf93bbcd5, State: Running, Role: LEADER
I20260929 21:04:24.723826 19732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3798ac8e0aa64da58cad6ecdf93bbcd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3798ac8e0aa64da58cad6ecdf93bbcd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39617 } }
I20260929 21:04:24.724336 19727 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3798ac8e0aa64da58cad6ecdf93bbcd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:04:24.727370 19734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3798ac8e0aa64da58cad6ecdf93bbcd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3798ac8e0aa64da58cad6ecdf93bbcd5. Latest consensus state: current_term: 1 leader_uuid: "3798ac8e0aa64da58cad6ecdf93bbcd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3798ac8e0aa64da58cad6ecdf93bbcd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39617 } } }
I20260929 21:04:24.727340 19733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3798ac8e0aa64da58cad6ecdf93bbcd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3798ac8e0aa64da58cad6ecdf93bbcd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3798ac8e0aa64da58cad6ecdf93bbcd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39617 } } }
I20260929 21:04:24.727815 19734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3798ac8e0aa64da58cad6ecdf93bbcd5 [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:24.727895 19733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3798ac8e0aa64da58cad6ecdf93bbcd5 [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:24.734058 19740 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:04:24.739334 19740 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:04:24.749014 19740 catalog_manager.cc:1347] Generated new cluster ID: e9a6da592cd241bab73bd2d302e07d82
I20260929 21:04:24.749174 19740 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:04:24.766696 19740 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:04:24.768571 19740 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:04:24.776862 19740 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3798ac8e0aa64da58cad6ecdf93bbcd5: Generated new TSK 0
I20260929 21:04:24.777932 19740 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 21:04:24.836632 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39617
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:04:25.162806 19753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:25.163133 19753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:25.163309 19753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:25.170761 19753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:25.170913 19753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:25.171098 19753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 21:04:25.181015 19753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:25.182348 19753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:25.183712 19753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:25.197052 19761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:25.197106 19759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:25.197149 19758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:25.198027 19753 server_base.cc:1034] running on GCE node
I20260929 21:04:25.198582 19753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:25.199467 19753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:25.200680 19753 hybrid_clock.cc:648] HybridClock initialized: now 1790715865200639 us; error 38 us; skew 500 ppm
I20260929 21:04:25.201051 19753 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:04:25 dist-test-slave-1dms krb5kdc[19647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:04:25.204164 19753 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 21:04:25.205600 19753 webserver.cc:460] Webserver started at http://127.0.97.1:41353/ using document root <none> and password file <none>
I20260929 21:04:25.206265 19753 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:25.206408 19753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:25.206727 19753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:25.209600 19753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-0/data/instance:
uuid: "8f79d27fc0b8408cae4c139bfb697114"
format_stamp: "Formatted at 2026-09-29 21:04:25 on dist-test-slave-1dms"
server_key: "120556c54d77ea3eeff3753d754d0222"
server_key_iv: "3cb93c83efb5e4249bce432450a0489c"
server_key_version: "encryptionkey@0"
I20260929 21:04:25.210356 19753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8f79d27fc0b8408cae4c139bfb697114"
format_stamp: "Formatted at 2026-09-29 21:04:25 on dist-test-slave-1dms"
server_key: "120556c54d77ea3eeff3753d754d0222"
server_key_iv: "3cb93c83efb5e4249bce432450a0489c"
server_key_version: "encryptionkey@0"
I20260929 21:04:25.215181 19753 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 21:04:25.218464 19768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:25.219676 19753 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 21:04:25.219846 19753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-0/wal
uuid: "8f79d27fc0b8408cae4c139bfb697114"
format_stamp: "Formatted at 2026-09-29 21:04:25 on dist-test-slave-1dms"
server_key: "120556c54d77ea3eeff3753d754d0222"
server_key_iv: "3cb93c83efb5e4249bce432450a0489c"
server_key_version: "encryptionkey@0"
I20260929 21:04:25.220043 19753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:25.239912 19753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:25.244360 19753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:25.244685 19753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:25.245723 19753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:04:25.247031 19753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:04:25.247143 19753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:25.247301 19753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:04:25.247375 19753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:25.279896 19753 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42117
I20260929 21:04:25.279945 19880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42117 every 8 connection(s)
I20260929 21:04:25.281275 19753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
I20260929 21:04:25.282724 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 19753
I20260929 21:04:25.282907 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-0/wal/instance
I20260929 21:04:25.283320 388 external_mini_cluster.cc:1428] Setting key 382f7cef675dc014c5d95f175f672808
Sep 29 21:04:25 dist-test-slave-1dms krb5kdc[19647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:04:25.303013 19883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:25.283896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38807 (local address 127.0.97.62:39617)
0929 21:04:25.284363 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:25.284370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:25.285555 (+ 1185us) server_negotiation.cc:407] Connection header received
0929 21:04:25.288455 (+ 2900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:25.288460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:25.288525 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:25.288673 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:25.290277 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:25.291124 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:25.292816 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:25.293244 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:25.293386 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:25.297176 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:25.297182 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:25.297185 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:04:25.297218 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:25.299860 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:25.300635 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:25.300640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:25.300643 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:25.300765 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:25.301240 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:25.301244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:25.301246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:25.301511 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:25.301692 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:25.302399 (+ 707us) server_negotiation.cc:299] Negotiation successful
0929 21:04:25.302651 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":230,"threads_started":1}
I20260929 21:04:25.304001 19884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:25.284527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.1:38807)
0929 21:04:25.285399 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:25.285436 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:25.288158 (+ 2722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:25.288995 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:25.289003 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:25.289531 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:04:25.290104 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:25.290124 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:25.291288 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:25.291300 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:25.292678 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:25.292690 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:25.293483 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:25.293492 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:25.293744 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:25.294455 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:04:25.294482 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:04:25.296947 (+ 2465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:04:25.300030 (+ 3083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:04:25.300039 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:04:25.300056 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:04:25.300483 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:04:25.300906 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:04:25.300910 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:04:25.300914 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:04:25.301112 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:04:25.301661 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:04:25.301669 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:04:25.302012 (+ 343us) client_negotiation.cc:769] Sending connection context
0929 21:04:25.302289 (+ 277us) client_negotiation.cc:240] Negotiation successful
0929 21:04:25.302819 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":343,"threads_started":1}
I20260929 21:04:25.305707 19881 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39617
I20260929 21:04:25.306011 19881 heartbeater.cc:464] Registering TS with master...
I20260929 21:04:25.306740 19881 heartbeater.cc:511] Master 127.0.97.62:39617 requested a full tablet report, sending...
I20260929 21:04:25.309166 19692 ts_manager.cc:194] Registered new tserver with Master: 8f79d27fc0b8408cae4c139bfb697114 (127.0.97.1:42117)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 21:04:25.310849 19692 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:38807
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 21:04:25.340332 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39617
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:04:25.668242 19887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:25.668527 19887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:25.668687 19887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:25.676000 19887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:25.676107 19887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:25.676316 19887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 21:04:25.686133 19887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:25.687366 19887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:25.688651 19887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:25.702169 19895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:25.702163 19893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:25.702559 19892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:25.704028 19887 server_base.cc:1034] running on GCE node
I20260929 21:04:25.704911 19887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:25.706187 19887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:25.707440 19887 hybrid_clock.cc:648] HybridClock initialized: now 1790715865707383 us; error 45 us; skew 500 ppm
I20260929 21:04:25.707842 19887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:04:25 dist-test-slave-1dms krb5kdc[19647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:04:25.711593 19887 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 21:04:25.713074 19887 webserver.cc:460] Webserver started at http://127.0.97.2:43241/ using document root <none> and password file <none>
I20260929 21:04:25.713738 19887 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:25.713832 19887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:25.714083 19887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:25.716894 19887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-1/data/instance:
uuid: "3bb906e0ed4f40378ae19d80b911e4cd"
format_stamp: "Formatted at 2026-09-29 21:04:25 on dist-test-slave-1dms"
server_key: "9d9edd168ba6f14f7a08ad967c6fc5fc"
server_key_iv: "9041874ebacf1aaa85e1c80ec0555271"
server_key_version: "encryptionkey@0"
I20260929 21:04:25.717643 19887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3bb906e0ed4f40378ae19d80b911e4cd"
format_stamp: "Formatted at 2026-09-29 21:04:25 on dist-test-slave-1dms"
server_key: "9d9edd168ba6f14f7a08ad967c6fc5fc"
server_key_iv: "9041874ebacf1aaa85e1c80ec0555271"
server_key_version: "encryptionkey@0"
I20260929 21:04:25.722513 19887 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260929 21:04:25.726280 19902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:25.727634 19887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 21:04:25.727761 19887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-1/wal
uuid: "3bb906e0ed4f40378ae19d80b911e4cd"
format_stamp: "Formatted at 2026-09-29 21:04:25 on dist-test-slave-1dms"
server_key: "9d9edd168ba6f14f7a08ad967c6fc5fc"
server_key_iv: "9041874ebacf1aaa85e1c80ec0555271"
server_key_version: "encryptionkey@0"
I20260929 21:04:25.727916 19887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:25.750926 19887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:25.755246 19887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:25.755491 19887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:25.756356 19887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:04:25.757684 19887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:04:25.757756 19887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:25.757836 19887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:04:25.757879 19887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:25.791118 19887 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44071
I20260929 21:04:25.791170 20014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44071 every 8 connection(s)
I20260929 21:04:25.792544 19887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
I20260929 21:04:25.793959 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 19887
I20260929 21:04:25.794162 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-1/wal/instance
I20260929 21:04:25.794718 388 external_mini_cluster.cc:1428] Setting key b7b4f73ca18cdb65502287bc5645efd6
Sep 29 21:04:25 dist-test-slave-1dms krb5kdc[19647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:04:25.816392 19883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:25.795444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53409 (local address 127.0.97.62:39617)
0929 21:04:25.795614 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:25.795618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:25.796990 (+ 1372us) server_negotiation.cc:407] Connection header received
0929 21:04:25.799928 (+ 2938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:25.799932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:25.799994 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:25.800081 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:25.802121 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:25.802830 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:25.805334 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:25.805788 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:25.805887 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:25.809567 (+ 3680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:25.809575 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:25.809583 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:04:25.809629 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:25.813176 (+ 3547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:25.813921 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:25.813929 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:25.813935 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:25.814096 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:25.814494 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:25.814502 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:25.814508 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:25.814896 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:25.815045 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:25.815684 (+ 639us) server_negotiation.cc:299] Negotiation successful
0929 21:04:25.815946 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 21:04:25.817044 20017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:25.795956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.2:53409)
0929 21:04:25.796781 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:25.796869 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:25.799614 (+ 2745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:25.800344 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:25.800359 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:25.801144 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:04:25.801928 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:25.801953 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:25.802992 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:25.803000 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:25.805188 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:25.805206 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:25.806015 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:25.806024 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:25.806339 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:25.807053 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:04:25.807076 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:04:25.809318 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:04:25.813321 (+ 4003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:04:25.813327 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:04:25.813340 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:04:25.813779 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:04:25.814207 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:04:25.814210 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:04:25.814213 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:04:25.814375 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:04:25.815012 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:04:25.815017 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:04:25.815307 (+ 290us) client_negotiation.cc:769] Sending connection context
0929 21:04:25.815522 (+ 215us) client_negotiation.cc:240] Negotiation successful
0929 21:04:25.815897 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":313,"threads_started":1}
I20260929 21:04:25.818605 20015 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39617
I20260929 21:04:25.818918 20015 heartbeater.cc:464] Registering TS with master...
I20260929 21:04:25.819597 20015 heartbeater.cc:511] Master 127.0.97.62:39617 requested a full tablet report, sending...
I20260929 21:04:25.821065 19692 ts_manager.cc:194] Registered new tserver with Master: 3bb906e0ed4f40378ae19d80b911e4cd (127.0.97.2:44071)
I20260929 21:04:25.821853 19692 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53409
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 21:04:25.851996 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39617
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:04:26.173559 20020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:26.173828 20020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:26.174000 20020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:26.181329 20020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:26.181438 20020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:26.181631 20020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 21:04:26.191242 20020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:26.192615 20020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:26.193930 20020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:26.206780 20028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:26.206873 20025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:26.206786 20026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:26.207335 20020 server_base.cc:1034] running on GCE node
I20260929 21:04:26.207914 20020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:26.208722 20020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:26.209918 20020 hybrid_clock.cc:648] HybridClock initialized: now 1790715866209861 us; error 54 us; skew 500 ppm
I20260929 21:04:26.210219 20020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:04:26 dist-test-slave-1dms krb5kdc[19647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:04:26.213229 20020 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 21:04:26.214632 20020 webserver.cc:460] Webserver started at http://127.0.97.3:46101/ using document root <none> and password file <none>
I20260929 21:04:26.215298 20020 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:26.215440 20020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:26.215745 20020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:26.218550 20020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-2/data/instance:
uuid: "14e3eb8fb5dc476c8f1a6c6186000ce6"
format_stamp: "Formatted at 2026-09-29 21:04:26 on dist-test-slave-1dms"
server_key: "3ce2f27dac69377b955491d99869a674"
server_key_iv: "c16054511ac994d44e80b43de1630d12"
server_key_version: "encryptionkey@0"
I20260929 21:04:26.219311 20020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-2/wal/instance:
uuid: "14e3eb8fb5dc476c8f1a6c6186000ce6"
format_stamp: "Formatted at 2026-09-29 21:04:26 on dist-test-slave-1dms"
server_key: "3ce2f27dac69377b955491d99869a674"
server_key_iv: "c16054511ac994d44e80b43de1630d12"
server_key_version: "encryptionkey@0"
I20260929 21:04:26.224191 20020 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 21:04:26.227792 20035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:26.229084 20020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:04:26.229254 20020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-2/wal
uuid: "14e3eb8fb5dc476c8f1a6c6186000ce6"
format_stamp: "Formatted at 2026-09-29 21:04:26 on dist-test-slave-1dms"
server_key: "3ce2f27dac69377b955491d99869a674"
server_key_iv: "c16054511ac994d44e80b43de1630d12"
server_key_version: "encryptionkey@0"
I20260929 21:04:26.229444 20020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:26.246696 20020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:26.250671 20020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:26.250957 20020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:26.251884 20020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:04:26.253228 20020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:04:26.253341 20020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:26.253494 20020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:04:26.253566 20020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:26.286120 20020 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36597
I20260929 21:04:26.286139 20147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36597 every 8 connection(s)
I20260929 21:04:26.287627 20020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
I20260929 21:04:26.294665 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 20020
I20260929 21:04:26.294858 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-2/wal/instance
I20260929 21:04:26.295284 388 external_mini_cluster.cc:1428] Setting key 16c8d85786431d51bf7ebbf3b2438c5e
Sep 29 21:04:26 dist-test-slave-1dms krb5kdc[19647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:04:26.306815 19883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:26.290264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34369 (local address 127.0.97.62:39617)
0929 21:04:26.290443 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:26.290447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:26.291506 (+ 1059us) server_negotiation.cc:407] Connection header received
0929 21:04:26.293456 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:26.293462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:26.293554 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:26.293674 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:26.295493 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:26.296303 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:26.297986 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:26.298468 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:26.298566 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:26.301756 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:26.301760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:26.301763 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:04:26.301796 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:26.304203 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:26.304890 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:26.304896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:26.304901 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:26.305034 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:26.305400 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:26.305403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:26.305405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:26.305657 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:26.305774 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:26.306388 (+ 614us) server_negotiation.cc:299] Negotiation successful
0929 21:04:26.306568 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 21:04:26.307729 20150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:26.290679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.3:34369)
0929 21:04:26.291350 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:26.291390 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:26.293186 (+ 1796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:26.293897 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:26.293905 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:26.294592 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:04:26.295264 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:26.295282 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:26.296451 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:26.296456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:26.297858 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:26.297872 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:26.298653 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:26.298659 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:26.298839 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:26.299503 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:04:26.299528 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:04:26.301582 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:04:26.304322 (+ 2740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:04:26.304329 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:04:26.304341 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:04:26.304745 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:04:26.305127 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:04:26.305130 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:04:26.305133 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:04:26.305304 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:04:26.305760 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:04:26.305765 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:04:26.306059 (+ 294us) client_negotiation.cc:769] Sending connection context
0929 21:04:26.306280 (+ 221us) client_negotiation.cc:240] Negotiation successful
0929 21:04:26.306645 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":270,"threads_started":1}
I20260929 21:04:26.309202 20148 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39617
I20260929 21:04:26.309516 20148 heartbeater.cc:464] Registering TS with master...
I20260929 21:04:26.310278 20148 heartbeater.cc:511] Master 127.0.97.62:39617 requested a full tablet report, sending...
I20260929 21:04:26.311599 19692 ts_manager.cc:194] Registered new tserver with Master: 14e3eb8fb5dc476c8f1a6c6186000ce6 (127.0.97.3:36597)
I20260929 21:04:26.312372 19692 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34369
I20260929 21:04:26.313944 19881 heartbeater.cc:503] Master 127.0.97.62:39617 was elected leader, sending a full tablet report...
I20260929 21:04:26.322278 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 21:04:26.339124 19883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:26.326333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36444 (local address 127.0.97.62:39617)
0929 21:04:26.326520 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:26.326529 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:26.326910 (+ 381us) server_negotiation.cc:407] Connection header received
0929 21:04:26.327099 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:26.327104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:26.327177 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:26.327277 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:26.328639 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:26.329403 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:26.330736 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:26.331224 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:26.331358 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:26.333991 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:26.333997 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:26.334000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:04:26.334031 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:26.336460 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:26.337176 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:26.337183 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:26.337191 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:26.337304 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:26.337675 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:26.337683 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:26.337686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:26.338009 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:26.338278 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:26.338598 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 21:04:26.338824 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 21:04:26.342572 19692 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 21:04:26.349311 19692 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 21:04:26.376310 20160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:26.360869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44071 (local address 127.0.0.1:46624)
0929 21:04:26.364302 (+ 3433us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:26.364321 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:26.364439 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:26.365474 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:26.365482 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:26.365517 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:04:26.366056 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:26.366071 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:26.368562 (+ 2491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:26.368571 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:26.371977 (+ 3406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:26.371998 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:26.374879 (+ 2881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:26.374889 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:26.375169 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:26.375242 (+ 73us) client_negotiation.cc:769] Sending connection context
0929 21:04:26.375403 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 21:04:26.375658 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3307,"mutex_wait_us":121,"thread_start_us":1867,"threads_started":1}
I20260929 21:04:26.377557 20163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:26.361426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46624 (local address 127.0.97.2:44071)
0929 21:04:26.363864 (+ 2438us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:26.363876 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:26.364561 (+ 685us) server_negotiation.cc:407] Connection header received
0929 21:04:26.364673 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:26.364682 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:26.364983 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:26.365196 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:04:26.366309 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:26.368357 (+ 2048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:26.372219 (+ 3862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:26.374260 (+ 2041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:26.374424 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:26.374704 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:26.376882 (+ 2178us) server_negotiation.cc:299] Negotiation successful
0929 21:04:26.377135 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2238,"thread_start_us":2105,"threads_started":1}
I20260929 21:04:26.378258 20161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:26.360646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36597 (local address 127.0.0.1:35254)
0929 21:04:26.362581 (+ 1935us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:26.362635 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:26.362866 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:26.364520 (+ 1654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:26.364529 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:26.364568 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:04:26.364965 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:26.364982 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:26.368928 (+ 3946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:26.368937 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:26.373177 (+ 4240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:26.373196 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:26.375971 (+ 2775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:26.375981 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:26.377393 (+ 1412us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:26.377448 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 21:04:26.377547 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 21:04:26.377715 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1630,"mutex_wait_us":1,"thread_start_us":3394,"threads_started":1}
I20260929 21:04:26.379837 20162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:26.360955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35254 (local address 127.0.97.3:36597)
0929 21:04:26.361935 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:26.361948 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:26.363119 (+ 1171us) server_negotiation.cc:407] Connection header received
0929 21:04:26.363241 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:26.363253 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:26.364048 (+ 795us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:26.364240 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:04:26.365232 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:26.368675 (+ 3443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:26.373406 (+ 4731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:26.375346 (+ 1940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:26.375502 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:26.375782 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:26.379332 (+ 3550us) server_negotiation.cc:299] Negotiation successful
0929 21:04:26.379497 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":768,"thread_start_us":615,"threads_started":1}
I20260929 21:04:26.381913 19950 tablet_service.cc:1467] Processing CreateTablet for tablet 78a7f79fadf44fa290935db4628091ed (DEFAULT_TABLE table=test-table [id=59a2c1a8915348338759dea6d2128a52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:04:26.383920 19950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78a7f79fadf44fa290935db4628091ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:04:26.384716 20083 tablet_service.cc:1467] Processing CreateTablet for tablet 78a7f79fadf44fa290935db4628091ed (DEFAULT_TABLE table=test-table [id=59a2c1a8915348338759dea6d2128a52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:04:26.386132 20083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78a7f79fadf44fa290935db4628091ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:04:26.386767 20161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:26.364621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42117 (local address 127.0.0.1:40008)
0929 21:04:26.378488 (+ 13867us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:26.378511 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:26.378637 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:26.380221 (+ 1584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:26.380225 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:26.380249 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:04:26.380429 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:26.380439 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:26.382305 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:26.382310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:26.384121 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:26.384135 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:26.385959 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:26.385966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:26.386138 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:26.386173 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 21:04:26.386270 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 21:04:26.386403 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13766,"thread_start_us":15858,"threads_started":1}
I20260929 21:04:26.387477 20165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:26.364790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40008 (local address 127.0.97.1:42117)
0929 21:04:26.379328 (+ 14538us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:26.379338 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:26.379368 (+ 30us) server_negotiation.cc:407] Connection header received
0929 21:04:26.379462 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:26.379471 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:26.379729 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:26.379911 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:04:26.380601 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:26.382124 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:26.384285 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:26.385767 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:26.385904 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:26.386128 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:26.386993 (+ 865us) server_negotiation.cc:299] Negotiation successful
0929 21:04:26.387160 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14365,"thread_start_us":14150,"threads_started":1}
I20260929 21:04:26.391372 19816 tablet_service.cc:1467] Processing CreateTablet for tablet 78a7f79fadf44fa290935db4628091ed (DEFAULT_TABLE table=test-table [id=59a2c1a8915348338759dea6d2128a52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:04:26.392760 19816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78a7f79fadf44fa290935db4628091ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:04:26.397554 20166 tablet_bootstrap.cc:492] T 78a7f79fadf44fa290935db4628091ed P 14e3eb8fb5dc476c8f1a6c6186000ce6: Bootstrap starting.
I20260929 21:04:26.399494 20167 tablet_bootstrap.cc:492] T 78a7f79fadf44fa290935db4628091ed P 3bb906e0ed4f40378ae19d80b911e4cd: Bootstrap starting.
I20260929 21:04:26.402643 20166 tablet_bootstrap.cc:654] T 78a7f79fadf44fa290935db4628091ed P 14e3eb8fb5dc476c8f1a6c6186000ce6: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:26.402711 20167 tablet_bootstrap.cc:654] T 78a7f79fadf44fa290935db4628091ed P 3bb906e0ed4f40378ae19d80b911e4cd: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:26.403913 20167 log.cc:824] T 78a7f79fadf44fa290935db4628091ed P 3bb906e0ed4f40378ae19d80b911e4cd: Log is configured to *not* fsync() on all Append() calls
I20260929 21:04:26.403913 20166 log.cc:824] T 78a7f79fadf44fa290935db4628091ed P 14e3eb8fb5dc476c8f1a6c6186000ce6: Log is configured to *not* fsync() on all Append() calls
I20260929 21:04:26.405848 20170 tablet_bootstrap.cc:492] T 78a7f79fadf44fa290935db4628091ed P 8f79d27fc0b8408cae4c139bfb697114: Bootstrap starting.
I20260929 21:04:26.407116 20166 tablet_bootstrap.cc:492] T 78a7f79fadf44fa290935db4628091ed P 14e3eb8fb5dc476c8f1a6c6186000ce6: No bootstrap required, opened a new log
I20260929 21:04:26.407382 20166 ts_tablet_manager.cc:1397] T 78a7f79fadf44fa290935db4628091ed P 14e3eb8fb5dc476c8f1a6c6186000ce6: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260929 21:04:26.407430 20167 tablet_bootstrap.cc:492] T 78a7f79fadf44fa290935db4628091ed P 3bb906e0ed4f40378ae19d80b911e4cd: No bootstrap required, opened a new log
I20260929 21:04:26.407658 20167 ts_tablet_manager.cc:1397] T 78a7f79fadf44fa290935db4628091ed P 3bb906e0ed4f40378ae19d80b911e4cd: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 21:04:26.408870 20170 tablet_bootstrap.cc:654] T 78a7f79fadf44fa290935db4628091ed P 8f79d27fc0b8408cae4c139bfb697114: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:26.410066 20170 log.cc:824] T 78a7f79fadf44fa290935db4628091ed P 8f79d27fc0b8408cae4c139bfb697114: Log is configured to *not* fsync() on all Append() calls
I20260929 21:04:26.412457 20170 tablet_bootstrap.cc:492] T 78a7f79fadf44fa290935db4628091ed P 8f79d27fc0b8408cae4c139bfb697114: No bootstrap required, opened a new log
I20260929 21:04:26.412709 20170 ts_tablet_manager.cc:1397] T 78a7f79fadf44fa290935db4628091ed P 8f79d27fc0b8408cae4c139bfb697114: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 21:04:26.418166 20166 raft_consensus.cc:348] T 78a7f79fadf44fa290935db4628091ed P 14e3eb8fb5dc476c8f1a6c6186000ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e3eb8fb5dc476c8f1a6c6186000ce6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36597 } } peers { permanent_uuid: "3bb906e0ed4f40378ae19d80b911e4cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44071 } } peers { permanent_uuid: "8f79d27fc0b8408cae4c139bfb697114" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42117 } }
I20260929 21:04:26.418622 20166 raft_consensus.cc:374] T 78a7f79fadf44fa290935db4628091ed P 14e3eb8fb5dc476c8f1a6c6186000ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:26.418747 20166 raft_consensus.cc:729] T 78a7f79fadf44fa290935db4628091ed P 14e3eb8fb5dc476c8f1a6c6186000ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14e3eb8fb5dc476c8f1a6c6186000ce6, State: Initialized, Role: FOLLOWER
I20260929 21:04:26.418709 20167 raft_consensus.cc:348] T 78a7f79fadf44fa290935db4628091ed P 3bb906e0ed4f40378ae19d80b911e4cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e3eb8fb5dc476c8f1a6c6186000ce6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36597 } } peers { permanent_uuid: "3bb906e0ed4f40378ae19d80b911e4cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44071 } } peers { permanent_uuid: "8f79d27fc0b8408cae4c139bfb697114" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42117 } }
I20260929 21:04:26.419221 20167 raft_consensus.cc:374] T 78a7f79fadf44fa290935db4628091ed P 3bb906e0ed4f40378ae19d80b911e4cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:26.419325 20167 raft_consensus.cc:729] T 78a7f79fadf44fa290935db4628091ed P 3bb906e0ed4f40378ae19d80b911e4cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bb906e0ed4f40378ae19d80b911e4cd, State: Initialized, Role: FOLLOWER
I20260929 21:04:26.419389 20166 consensus_queue.cc:260] T 78a7f79fadf44fa290935db4628091ed P 14e3eb8fb5dc476c8f1a6c6186000ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e3eb8fb5dc476c8f1a6c6186000ce6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36597 } } peers { permanent_uuid: "3bb906e0ed4f40378ae19d80b911e4cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44071 } } peers { permanent_uuid: "8f79d27fc0b8408cae4c139bfb697114" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42117 } }
I20260929 21:04:26.419903 20167 consensus_queue.cc:260] T 78a7f79fadf44fa290935db4628091ed P 3bb906e0ed4f40378ae19d80b911e4cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e3eb8fb5dc476c8f1a6c6186000ce6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36597 } } peers { permanent_uuid: "3bb906e0ed4f40378ae19d80b911e4cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44071 } } peers { permanent_uuid: "8f79d27fc0b8408cae4c139bfb697114" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42117 } }
I20260929 21:04:26.419899 20170 raft_consensus.cc:348] T 78a7f79fadf44fa290935db4628091ed P 8f79d27fc0b8408cae4c139bfb697114 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e3eb8fb5dc476c8f1a6c6186000ce6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36597 } } peers { permanent_uuid: "3bb906e0ed4f40378ae19d80b911e4cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44071 } } peers { permanent_uuid: "8f79d27fc0b8408cae4c139bfb697114" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42117 } }
I20260929 21:04:26.420395 20170 raft_consensus.cc:374] T 78a7f79fadf44fa290935db4628091ed P 8f79d27fc0b8408cae4c139bfb697114 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:26.420504 20170 raft_consensus.cc:729] T 78a7f79fadf44fa290935db4628091ed P 8f79d27fc0b8408cae4c139bfb697114 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f79d27fc0b8408cae4c139bfb697114, State: Initialized, Role: FOLLOWER
I20260929 21:04:26.420631 20148 heartbeater.cc:503] Master 127.0.97.62:39617 was elected leader, sending a full tablet report...
I20260929 21:04:26.421110 20170 consensus_queue.cc:260] T 78a7f79fadf44fa290935db4628091ed P 8f79d27fc0b8408cae4c139bfb697114 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e3eb8fb5dc476c8f1a6c6186000ce6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36597 } } peers { permanent_uuid: "3bb906e0ed4f40378ae19d80b911e4cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44071 } } peers { permanent_uuid: "8f79d27fc0b8408cae4c139bfb697114" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42117 } }
I20260929 21:04:26.423540 20170 ts_tablet_manager.cc:1428] T 78a7f79fadf44fa290935db4628091ed P 8f79d27fc0b8408cae4c139bfb697114: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260929 21:04:26.425729 20166 ts_tablet_manager.cc:1428] T 78a7f79fadf44fa290935db4628091ed P 14e3eb8fb5dc476c8f1a6c6186000ce6: Time spent starting tablet: real 0.018s user 0.016s sys 0.000s
I20260929 21:04:26.426365 20015 heartbeater.cc:503] Master 127.0.97.62:39617 was elected leader, sending a full tablet report...
I20260929 21:04:26.426983 20167 ts_tablet_manager.cc:1428] T 78a7f79fadf44fa290935db4628091ed P 3bb906e0ed4f40378ae19d80b911e4cd: Time spent starting tablet: real 0.019s user 0.013s sys 0.004s
I20260929 21:04:26.486584 20172 raft_consensus.cc:482] T 78a7f79fadf44fa290935db4628091ed P 14e3eb8fb5dc476c8f1a6c6186000ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 21:04:26.486927 20172 raft_consensus.cc:504] T 78a7f79fadf44fa290935db4628091ed P 14e3eb8fb5dc476c8f1a6c6186000ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e3eb8fb5dc476c8f1a6c6186000ce6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36597 } } peers { permanent_uuid: "3bb906e0ed4f40378ae19d80b911e4cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44071 } } peers { permanent_uuid: "8f79d27fc0b8408cae4c139bfb697114" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42117 } }
I20260929 21:04:26.488608 20172 leader_election.cc:288] T 78a7f79fadf44fa290935db4628091ed P 14e3eb8fb5dc476c8f1a6c6186000ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3bb906e0ed4f40378ae19d80b911e4cd (127.0.97.2:44071), 8f79d27fc0b8408cae4c139bfb697114 (127.0.97.1:42117)
I20260929 21:04:26.494591 20163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:26.489202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55535 (local address 127.0.97.2:44071)
0929 21:04:26.489410 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:26.489418 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:26.489437 (+ 19us) server_negotiation.cc:407] Connection header received
0929 21:04:26.489526 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:26.489533 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:26.489617 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:26.489718 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:04:26.490142 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:26.491100 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:26.492980 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:26.493723 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:26.493869 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:26.493959 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:26.494220 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 21:04:26.494342 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 21:04:26.494618 20150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:26.488915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44071 (local address 127.0.97.3:55535)
0929 21:04:26.489153 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:26.489174 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:26.489300 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:26.489761 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:26.489771 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:26.489795 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:04:26.489980 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:26.489990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:26.491222 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:26.491228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:26.492779 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:26.492791 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:26.493877 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:26.493882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:26.494063 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:26.494087 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 21:04:26.494194 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 21:04:26.494318 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":42}
I20260929 21:04:26.495213 20165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:26.489275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33325 (local address 127.0.97.1:42117)
0929 21:04:26.489529 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:26.489538 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:26.489999 (+ 461us) server_negotiation.cc:407] Connection header received
0929 21:04:26.490121 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:26.490133 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:26.490197 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:26.490263 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:04:26.490830 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:26.491755 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:26.493536 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:26.494304 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:26.494454 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:26.494554 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:26.494936 (+ 382us) server_negotiation.cc:299] Negotiation successful
0929 21:04:26.495046 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 21:04:26.495836 20175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:26.488982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42117 (local address 127.0.97.3:33325)
0929 21:04:26.489892 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:26.489909 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:26.490005 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:26.490306 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:26.490309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:26.490332 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:04:26.490684 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:26.490696 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:26.491897 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:26.491901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:26.493381 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:26.493393 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:26.494517 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:26.494523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:26.494716 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:26.494757 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 21:04:26.495410 (+ 653us) client_negotiation.cc:240] Negotiation successful
0929 21:04:26.495536 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"mutex_wait_us":1,"thread_start_us":552,"threads_started":1}
I20260929 21:04:26.495687 19970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78a7f79fadf44fa290935db4628091ed" candidate_uuid: "14e3eb8fb5dc476c8f1a6c6186000ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bb906e0ed4f40378ae19d80b911e4cd" is_pre_election: true
I20260929 21:04:26.496194 19970 raft_consensus.cc:2454] T 78a7f79fadf44fa290935db4628091ed P 3bb906e0ed4f40378ae19d80b911e4cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14e3eb8fb5dc476c8f1a6c6186000ce6 in term 0.
I20260929 21:04:26.496812 20039 leader_election.cc:302] T 78a7f79fadf44fa290935db4628091ed P 14e3eb8fb5dc476c8f1a6c6186000ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14e3eb8fb5dc476c8f1a6c6186000ce6, 3bb906e0ed4f40378ae19d80b911e4cd; no voters:
I20260929 21:04:26.496708 19836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78a7f79fadf44fa290935db4628091ed" candidate_uuid: "14e3eb8fb5dc476c8f1a6c6186000ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f79d27fc0b8408cae4c139bfb697114" is_pre_election: true
I20260929 21:04:26.497257 20172 raft_consensus.cc:2781] T 78a7f79fadf44fa290935db4628091ed P 14e3eb8fb5dc476c8f1a6c6186000ce6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 21:04:26.497311 19836 raft_consensus.cc:2454] T 78a7f79fadf44fa290935db4628091ed P 8f79d27fc0b8408cae4c139bfb697114 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14e3eb8fb5dc476c8f1a6c6186000ce6 in term 0.
I20260929 21:04:26.497407 20172 raft_consensus.cc:482] T 78a7f79fadf44fa290935db4628091ed P 14e3eb8fb5dc476c8f1a6c6186000ce6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 21:04:26.497505 20172 raft_consensus.cc:3037] T 78a7f79fadf44fa290935db4628091ed P 14e3eb8fb5dc476c8f1a6c6186000ce6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:26.499802 20172 raft_consensus.cc:504] T 78a7f79fadf44fa290935db4628091ed P 14e3eb8fb5dc476c8f1a6c6186000ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e3eb8fb5dc476c8f1a6c6186000ce6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36597 } } peers { permanent_uuid: "3bb906e0ed4f40378ae19d80b911e4cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44071 } } peers { permanent_uuid: "8f79d27fc0b8408cae4c139bfb697114" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42117 } }
I20260929 21:04:26.500361 20172 leader_election.cc:288] T 78a7f79fadf44fa290935db4628091ed P 14e3eb8fb5dc476c8f1a6c6186000ce6 [CANDIDATE]: Term 1 election: Requested vote from peers 3bb906e0ed4f40378ae19d80b911e4cd (127.0.97.2:44071), 8f79d27fc0b8408cae4c139bfb697114 (127.0.97.1:42117)
I20260929 21:04:26.500676 19970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78a7f79fadf44fa290935db4628091ed" candidate_uuid: "14e3eb8fb5dc476c8f1a6c6186000ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bb906e0ed4f40378ae19d80b911e4cd"
I20260929 21:04:26.500795 19836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78a7f79fadf44fa290935db4628091ed" candidate_uuid: "14e3eb8fb5dc476c8f1a6c6186000ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f79d27fc0b8408cae4c139bfb697114"
I20260929 21:04:26.500909 19970 raft_consensus.cc:3037] T 78a7f79fadf44fa290935db4628091ed P 3bb906e0ed4f40378ae19d80b911e4cd [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:26.501056 19836 raft_consensus.cc:3037] T 78a7f79fadf44fa290935db4628091ed P 8f79d27fc0b8408cae4c139bfb697114 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:26.504482 19970 raft_consensus.cc:2454] T 78a7f79fadf44fa290935db4628091ed P 3bb906e0ed4f40378ae19d80b911e4cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14e3eb8fb5dc476c8f1a6c6186000ce6 in term 1.
I20260929 21:04:26.504762 19836 raft_consensus.cc:2454] T 78a7f79fadf44fa290935db4628091ed P 8f79d27fc0b8408cae4c139bfb697114 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14e3eb8fb5dc476c8f1a6c6186000ce6 in term 1.
I20260929 21:04:26.504958 20039 leader_election.cc:302] T 78a7f79fadf44fa290935db4628091ed P 14e3eb8fb5dc476c8f1a6c6186000ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14e3eb8fb5dc476c8f1a6c6186000ce6, 3bb906e0ed4f40378ae19d80b911e4cd; no voters:
I20260929 21:04:26.505312 20172 raft_consensus.cc:2781] T 78a7f79fadf44fa290935db4628091ed P 14e3eb8fb5dc476c8f1a6c6186000ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:04:26.505789 20172 raft_consensus.cc:686] T 78a7f79fadf44fa290935db4628091ed P 14e3eb8fb5dc476c8f1a6c6186000ce6 [term 1 LEADER]: Becoming Leader. State: Replica: 14e3eb8fb5dc476c8f1a6c6186000ce6, State: Running, Role: LEADER
I20260929 21:04:26.506336 20172 consensus_queue.cc:237] T 78a7f79fadf44fa290935db4628091ed P 14e3eb8fb5dc476c8f1a6c6186000ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e3eb8fb5dc476c8f1a6c6186000ce6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36597 } } peers { permanent_uuid: "3bb906e0ed4f40378ae19d80b911e4cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44071 } } peers { permanent_uuid: "8f79d27fc0b8408cae4c139bfb697114" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42117 } }
I20260929 21:04:26.511615 19692 catalog_manager.cc:5495] T 78a7f79fadf44fa290935db4628091ed P 14e3eb8fb5dc476c8f1a6c6186000ce6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14e3eb8fb5dc476c8f1a6c6186000ce6 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "14e3eb8fb5dc476c8f1a6c6186000ce6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e3eb8fb5dc476c8f1a6c6186000ce6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bb906e0ed4f40378ae19d80b911e4cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f79d27fc0b8408cae4c139bfb697114" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42117 } health_report { overall_health: UNKNOWN } } }
W20260929 21:04:26.533699 19882 tablet.cc:2369] T 78a7f79fadf44fa290935db4628091ed P 8f79d27fc0b8408cae4c139bfb697114: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 21:04:26.539237 20149 tablet.cc:2369] T 78a7f79fadf44fa290935db4628091ed P 14e3eb8fb5dc476c8f1a6c6186000ce6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 21:04:26.544625 20016 tablet.cc:2369] T 78a7f79fadf44fa290935db4628091ed P 3bb906e0ed4f40378ae19d80b911e4cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 21:04:26.560745 20162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:26.555976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35268 (local address 127.0.97.3:36597)
0929 21:04:26.556177 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:26.556182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:26.556262 (+ 80us) server_negotiation.cc:407] Connection header received
0929 21:04:26.556422 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:26.556425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:26.556508 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:26.556630 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:04:26.556998 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:26.557702 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:26.558996 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:26.559370 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:26.559458 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:26.559852 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:04:26.560327 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:04:26.560431 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:26.560496 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 21:04:26.560554 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 21:04:26.568575 19970 raft_consensus.cc:1261] T 78a7f79fadf44fa290935db4628091ed P 3bb906e0ed4f40378ae19d80b911e4cd [term 1 FOLLOWER]: Refusing update from remote peer 14e3eb8fb5dc476c8f1a6c6186000ce6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:04:26.568575 19836 raft_consensus.cc:1261] T 78a7f79fadf44fa290935db4628091ed P 8f79d27fc0b8408cae4c139bfb697114 [term 1 FOLLOWER]: Refusing update from remote peer 14e3eb8fb5dc476c8f1a6c6186000ce6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:04:26.569653 20176 consensus_queue.cc:1035] T 78a7f79fadf44fa290935db4628091ed P 14e3eb8fb5dc476c8f1a6c6186000ce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f79d27fc0b8408cae4c139bfb697114" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:04:26.570009 20172 consensus_queue.cc:1035] T 78a7f79fadf44fa290935db4628091ed P 14e3eb8fb5dc476c8f1a6c6186000ce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bb906e0ed4f40378ae19d80b911e4cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:04:26.580694 20179 mvcc.cc:205] Tried to move back new op lower bound from 7334772189454811136 to 7334772189214769152. Current Snapshot: MvccSnapshot[applied={T|T < 7334772189454811136}]
I20260929 21:04:26.595569 19691 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 21:04:26.595849 19691 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 21:04:26.599196 19691 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3798ac8e0aa64da58cad6ecdf93bbcd5: Sending DeleteTablet for 3 replicas of tablet 78a7f79fadf44fa290935db4628091ed
I20260929 21:04:26.600312 19950 tablet_service.cc:1514] Processing DeleteTablet for tablet 78a7f79fadf44fa290935db4628091ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:04:26 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46624
I20260929 21:04:26.600358 20083 tablet_service.cc:1514] Processing DeleteTablet for tablet 78a7f79fadf44fa290935db4628091ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:04:26 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35254
I20260929 21:04:26.601325 19816 tablet_service.cc:1514] Processing DeleteTablet for tablet 78a7f79fadf44fa290935db4628091ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:04:26 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40008
I20260929 21:04:26.601406 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 19753
I20260929 21:04:26.615073 20191 tablet_replica.cc:330] T 78a7f79fadf44fa290935db4628091ed P 8f79d27fc0b8408cae4c139bfb697114: stopping tablet replica
I20260929 21:04:26.615624 20191 raft_consensus.cc:2229] T 78a7f79fadf44fa290935db4628091ed P 8f79d27fc0b8408cae4c139bfb697114 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:04:26.617625 20191 raft_consensus.cc:2258] T 78a7f79fadf44fa290935db4628091ed P 8f79d27fc0b8408cae4c139bfb697114 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:04:26.620524 20189 tablet_replica.cc:330] T 78a7f79fadf44fa290935db4628091ed P 14e3eb8fb5dc476c8f1a6c6186000ce6: stopping tablet replica
I20260929 21:04:26.620959 20189 raft_consensus.cc:2229] T 78a7f79fadf44fa290935db4628091ed P 14e3eb8fb5dc476c8f1a6c6186000ce6 [term 1 LEADER]: Raft consensus shutting down.
I20260929 21:04:26.621605 20189 raft_consensus.cc:2258] T 78a7f79fadf44fa290935db4628091ed P 14e3eb8fb5dc476c8f1a6c6186000ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:04:26.622391 20191 ts_tablet_manager.cc:1905] T 78a7f79fadf44fa290935db4628091ed P 8f79d27fc0b8408cae4c139bfb697114: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:04:26.622458 20190 tablet_replica.cc:330] T 78a7f79fadf44fa290935db4628091ed P 3bb906e0ed4f40378ae19d80b911e4cd: stopping tablet replica
I20260929 21:04:26.622862 20190 raft_consensus.cc:2229] T 78a7f79fadf44fa290935db4628091ed P 3bb906e0ed4f40378ae19d80b911e4cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:04:26.623390 20190 raft_consensus.cc:2258] T 78a7f79fadf44fa290935db4628091ed P 3bb906e0ed4f40378ae19d80b911e4cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:04:26.624683 20189 ts_tablet_manager.cc:1905] T 78a7f79fadf44fa290935db4628091ed P 14e3eb8fb5dc476c8f1a6c6186000ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:04:26.625566 20190 ts_tablet_manager.cc:1905] T 78a7f79fadf44fa290935db4628091ed P 3bb906e0ed4f40378ae19d80b911e4cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:04:26.629817 20191 ts_tablet_manager.cc:1918] T 78a7f79fadf44fa290935db4628091ed P 8f79d27fc0b8408cae4c139bfb697114: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:04:26.629995 20191 log.cc:1196] T 78a7f79fadf44fa290935db4628091ed P 8f79d27fc0b8408cae4c139bfb697114: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-0/wal/wals/78a7f79fadf44fa290935db4628091ed
Sep 29 21:04:26 dist-test-slave-1dms krb5kdc[19647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 21:04:26.630518 20191 ts_tablet_manager.cc:1939] T 78a7f79fadf44fa290935db4628091ed P 8f79d27fc0b8408cae4c139bfb697114: Deleting consensus metadata
I20260929 21:04:26.632671 20190 ts_tablet_manager.cc:1918] T 78a7f79fadf44fa290935db4628091ed P 3bb906e0ed4f40378ae19d80b911e4cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:04:26.632697 19678 catalog_manager.cc:4841] TS 8f79d27fc0b8408cae4c139bfb697114 (127.0.97.1:42117): tablet 78a7f79fadf44fa290935db4628091ed (table test-table [id=59a2c1a8915348338759dea6d2128a52]) successfully deleted
I20260929 21:04:26.632916 20190 log.cc:1196] T 78a7f79fadf44fa290935db4628091ed P 3bb906e0ed4f40378ae19d80b911e4cd: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-1/wal/wals/78a7f79fadf44fa290935db4628091ed
I20260929 21:04:26.633486 20190 ts_tablet_manager.cc:1939] T 78a7f79fadf44fa290935db4628091ed P 3bb906e0ed4f40378ae19d80b911e4cd: Deleting consensus metadata
I20260929 21:04:26.635779 19679 catalog_manager.cc:4841] TS 3bb906e0ed4f40378ae19d80b911e4cd (127.0.97.2:44071): tablet 78a7f79fadf44fa290935db4628091ed (table test-table [id=59a2c1a8915348338759dea6d2128a52]) successfully deleted
I20260929 21:04:26.637298 20189 ts_tablet_manager.cc:1918] T 78a7f79fadf44fa290935db4628091ed P 14e3eb8fb5dc476c8f1a6c6186000ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:04:26.637488 20189 log.cc:1196] T 78a7f79fadf44fa290935db4628091ed P 14e3eb8fb5dc476c8f1a6c6186000ce6: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790715556572573-388-0/minicluster-data/ts-2/wal/wals/78a7f79fadf44fa290935db4628091ed
I20260929 21:04:26.637993 20189 ts_tablet_manager.cc:1939] T 78a7f79fadf44fa290935db4628091ed P 14e3eb8fb5dc476c8f1a6c6186000ce6: Deleting consensus metadata
I20260929 21:04:26.639341 19678 catalog_manager.cc:4841] TS 14e3eb8fb5dc476c8f1a6c6186000ce6 (127.0.97.3:36597): tablet 78a7f79fadf44fa290935db4628091ed (table test-table [id=59a2c1a8915348338759dea6d2128a52]) successfully deleted
I20260929 21:04:26.641059 20165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:26.603305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40020 (local address 127.0.97.1:42117)
0929 21:04:26.603529 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:26.603539 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:26.611867 (+ 8328us) server_negotiation.cc:407] Connection header received
0929 21:04:26.612296 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:26.612302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:26.612399 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:26.612471 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:26.615934 (+ 3463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:26.617088 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:26.619947 (+ 2859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:26.620636 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:26.620761 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:26.633939 (+ 13178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:26.633954 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:26.633973 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:04:26.634024 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:26.637969 (+ 3945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:26.638970 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:26.638979 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:26.638984 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:26.639141 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:26.639599 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:26.639603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:26.639605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:26.639947 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:26.640169 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:26.640518 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 21:04:26.640686 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 21:04:26.641812 19876 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:04:26.730216 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 19753
I20260929 21:04:26.754277 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 19887
Sep 29 21:04:26 dist-test-slave-1dms krb5kdc[19647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 21:04:26.771301 20163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:26.755172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46628 (local address 127.0.97.2:44071)
0929 21:04:26.755348 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:26.755356 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:26.755374 (+ 18us) server_negotiation.cc:407] Connection header received
0929 21:04:26.755497 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:26.755503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:26.755569 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:26.755661 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:26.757370 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:26.758110 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:26.759931 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:26.760317 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:26.760406 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:26.765543 (+ 5137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:26.765556 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:26.765570 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:04:26.765602 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:26.768312 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:26.769361 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:26.769368 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:26.769370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:26.769484 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:26.769898 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:26.769902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:26.769903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:26.770277 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:26.770494 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:26.770819 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 21:04:26.770973 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 21:04:26.771978 20010 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:04:26.860371 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 19887
I20260929 21:04:26.878239 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 20020
Sep 29 21:04:26 dist-test-slave-1dms krb5kdc[19647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 21:04:26.895618 20162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:26.878991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35282 (local address 127.0.97.3:36597)
0929 21:04:26.879181 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:26.879187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:26.879317 (+ 130us) server_negotiation.cc:407] Connection header received
0929 21:04:26.879524 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:26.879528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:26.879594 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:26.879729 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:26.881058 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:26.881935 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:26.883377 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:26.884114 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:26.884256 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:26.888495 (+ 4239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:26.888509 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:26.888534 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:04:26.888592 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:26.892551 (+ 3959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:26.893458 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:26.893467 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:26.893470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:26.893647 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:26.894030 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:26.894036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:26.894040 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:26.894481 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:26.894803 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:26.894958 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 21:04:26.895193 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 21:04:26.896420 20143 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:04:27.017246 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 20020
I20260929 21:04:27.036618 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 19660
I20260929 21:04:27.037714 19722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:04:27.146966 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 19660
2026-09-29T21:04:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4298 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T21:04:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:04:27Z Disabled control of system clock
I20260929 21:04:27.219184 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36891
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 21:04:27.542611 20200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:27.542871 20200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:27.542950 20200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:27.549953 20200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:04:27.550045 20200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:27.550082 20200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:27.550135 20200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:04:27.550179 20200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:04:27.558952 20200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36891
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:27.560105 20200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:27.561798 20200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:27.573436 20206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:27.573493 20208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:27.573520 20205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:27.574208 20200 server_base.cc:1034] running on GCE node
I20260929 21:04:27.574944 20200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:27.576287 20200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:27.577509 20200 hybrid_clock.cc:648] HybridClock initialized: now 1790715867577463 us; error 57 us; skew 500 ppm
I20260929 21:04:27.577880 20200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 21:04:27.580190 20200 webserver.cc:460] Webserver started at http://127.0.97.62:39673/ using document root <none> and password file <none>
I20260929 21:04:27.580935 20200 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:27.581063 20200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:27.581382 20200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:27.584198 20200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "d39758ce551745409923e8db9555586b"
format_stamp: "Formatted at 2026-09-29 21:04:27 on dist-test-slave-1dms"
server_key: "12c166e28e67833a85be127986e6b8f9"
server_key_iv: "1b22baf05b8654cf381ab79408ed50ea"
server_key_version: "encryptionkey@0"
I20260929 21:04:27.585042 20200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "d39758ce551745409923e8db9555586b"
format_stamp: "Formatted at 2026-09-29 21:04:27 on dist-test-slave-1dms"
server_key: "12c166e28e67833a85be127986e6b8f9"
server_key_iv: "1b22baf05b8654cf381ab79408ed50ea"
server_key_version: "encryptionkey@0"
I20260929 21:04:27.590024 20200 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 21:04:27.593308 20214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:27.594566 20200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 21:04:27.594759 20200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "d39758ce551745409923e8db9555586b"
format_stamp: "Formatted at 2026-09-29 21:04:27 on dist-test-slave-1dms"
server_key: "12c166e28e67833a85be127986e6b8f9"
server_key_iv: "1b22baf05b8654cf381ab79408ed50ea"
server_key_version: "encryptionkey@0"
I20260929 21:04:27.594966 20200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:27.616823 20200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:27.617691 20200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:27.617972 20200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:27.635898 20200 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36891
I20260929 21:04:27.635912 20265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36891 every 8 connection(s)
I20260929 21:04:27.637277 20200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:04:27.639303 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 20200
I20260929 21:04:27.639465 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:04:27.639889 388 external_mini_cluster.cc:1428] Setting key 38eb4cc8a44da910af943853accc92d3
I20260929 21:04:27.641469 20266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:04:27.654868 20269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:27.641774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43868 (local address 127.0.97.62:36891)
0929 21:04:27.642799 (+ 1025us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:27.642814 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:27.642864 (+ 50us) server_negotiation.cc:407] Connection header received
0929 21:04:27.644740 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:27.644766 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:27.645364 (+ 598us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:27.645818 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:27.646319 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:27.648342 (+ 2023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:27.649747 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:27.650769 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:27.650954 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:27.652064 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:27.652074 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:27.652096 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:27.652859 (+ 763us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:27.652958 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:27.652992 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:27.653100 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:27.653281 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 21:04:27.653625 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":320,"threads_started":1}
I20260929 21:04:27.656006 20266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d39758ce551745409923e8db9555586b: Bootstrap starting.
I20260929 21:04:27.659042 20266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d39758ce551745409923e8db9555586b: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:27.660038 20266 log.cc:824] T 00000000000000000000000000000000 P d39758ce551745409923e8db9555586b: Log is configured to *not* fsync() on all Append() calls
I20260929 21:04:27.662801 20266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d39758ce551745409923e8db9555586b: No bootstrap required, opened a new log
I20260929 21:04:27.669994 20266 raft_consensus.cc:348] T 00000000000000000000000000000000 P d39758ce551745409923e8db9555586b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d39758ce551745409923e8db9555586b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36891 } }
I20260929 21:04:27.670331 20266 raft_consensus.cc:374] T 00000000000000000000000000000000 P d39758ce551745409923e8db9555586b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:27.670446 20266 raft_consensus.cc:729] T 00000000000000000000000000000000 P d39758ce551745409923e8db9555586b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d39758ce551745409923e8db9555586b, State: Initialized, Role: FOLLOWER
I20260929 21:04:27.670965 20266 consensus_queue.cc:260] T 00000000000000000000000000000000 P d39758ce551745409923e8db9555586b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d39758ce551745409923e8db9555586b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36891 } }
I20260929 21:04:27.671192 20266 raft_consensus.cc:388] T 00000000000000000000000000000000 P d39758ce551745409923e8db9555586b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:04:27.671351 20266 raft_consensus.cc:482] T 00000000000000000000000000000000 P d39758ce551745409923e8db9555586b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:04:27.671510 20266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d39758ce551745409923e8db9555586b [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:27.673578 20266 raft_consensus.cc:504] T 00000000000000000000000000000000 P d39758ce551745409923e8db9555586b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d39758ce551745409923e8db9555586b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36891 } }
I20260929 21:04:27.674057 20266 leader_election.cc:302] T 00000000000000000000000000000000 P d39758ce551745409923e8db9555586b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d39758ce551745409923e8db9555586b; no voters:
I20260929 21:04:27.674611 20266 leader_election.cc:288] T 00000000000000000000000000000000 P d39758ce551745409923e8db9555586b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 21:04:27.674772 20271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d39758ce551745409923e8db9555586b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:04:27.675426 20271 raft_consensus.cc:686] T 00000000000000000000000000000000 P d39758ce551745409923e8db9555586b [term 1 LEADER]: Becoming Leader. State: Replica: d39758ce551745409923e8db9555586b, State: Running, Role: LEADER
I20260929 21:04:27.675974 20271 consensus_queue.cc:237] T 00000000000000000000000000000000 P d39758ce551745409923e8db9555586b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d39758ce551745409923e8db9555586b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36891 } }
I20260929 21:04:27.676564 20266 sys_catalog.cc:564] T 00000000000000000000000000000000 P d39758ce551745409923e8db9555586b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:04:27.679010 20272 sys_catalog.cc:455] T 00000000000000000000000000000000 P d39758ce551745409923e8db9555586b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d39758ce551745409923e8db9555586b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d39758ce551745409923e8db9555586b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36891 } } }
I20260929 21:04:27.679111 20273 sys_catalog.cc:455] T 00000000000000000000000000000000 P d39758ce551745409923e8db9555586b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d39758ce551745409923e8db9555586b. Latest consensus state: current_term: 1 leader_uuid: "d39758ce551745409923e8db9555586b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d39758ce551745409923e8db9555586b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36891 } } }
I20260929 21:04:27.679466 20272 sys_catalog.cc:458] T 00000000000000000000000000000000 P d39758ce551745409923e8db9555586b [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:27.679502 20273 sys_catalog.cc:458] T 00000000000000000000000000000000 P d39758ce551745409923e8db9555586b [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:27.682247 20276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:04:27.686038 20276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:04:27.695417 20276 catalog_manager.cc:1347] Generated new cluster ID: f85d5c7c055f4fe9a64a41986a7884e3
I20260929 21:04:27.695547 20276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:04:27.704641 20276 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:04:27.706450 20276 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:04:27.719514 20276 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d39758ce551745409923e8db9555586b: Generated new TSK 0
I20260929 21:04:27.720455 20276 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 21:04:27.730784 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36891
--builtin_ntp_servers=127.0.97.20:39079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 21:04:28.048319 20290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:28.048578 20290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:28.048729 20290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:28.055823 20290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:28.055917 20290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:28.056118 20290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 21:04:28.065035 20290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:28.066293 20290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:28.068185 20290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:28.081319 20295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:28.081363 20298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:28.081444 20296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:28.082087 20290 server_base.cc:1034] running on GCE node
I20260929 21:04:28.082720 20290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:28.083659 20290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:28.084908 20290 hybrid_clock.cc:648] HybridClock initialized: now 1790715868084845 us; error 52 us; skew 500 ppm
I20260929 21:04:28.085213 20290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 21:04:28.087589 20290 webserver.cc:460] Webserver started at http://127.0.97.1:36781/ using document root <none> and password file <none>
I20260929 21:04:28.088315 20290 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:28.088438 20290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:28.088758 20290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:28.091593 20290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-0/data/instance:
uuid: "564de47e367f46a3810ca2934a71db6e"
format_stamp: "Formatted at 2026-09-29 21:04:28 on dist-test-slave-1dms"
server_key: "71428078e8f0455bf59e41da6e184e86"
server_key_iv: "ed88e04e9926899427e9e2e8d8b28208"
server_key_version: "encryptionkey@0"
I20260929 21:04:28.092386 20290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-0/wal/instance:
uuid: "564de47e367f46a3810ca2934a71db6e"
format_stamp: "Formatted at 2026-09-29 21:04:28 on dist-test-slave-1dms"
server_key: "71428078e8f0455bf59e41da6e184e86"
server_key_iv: "ed88e04e9926899427e9e2e8d8b28208"
server_key_version: "encryptionkey@0"
I20260929 21:04:28.097390 20290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 21:04:28.100786 20304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:28.102072 20290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 21:04:28.102242 20290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-0/wal
uuid: "564de47e367f46a3810ca2934a71db6e"
format_stamp: "Formatted at 2026-09-29 21:04:28 on dist-test-slave-1dms"
server_key: "71428078e8f0455bf59e41da6e184e86"
server_key_iv: "ed88e04e9926899427e9e2e8d8b28208"
server_key_version: "encryptionkey@0"
I20260929 21:04:28.102455 20290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:28.119076 20290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:28.119849 20290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:28.120111 20290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:28.121129 20290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:04:28.122509 20290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:04:28.122633 20290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:28.122749 20290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:04:28.122831 20290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:28.155194 20290 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37733
I20260929 21:04:28.155258 20416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37733 every 8 connection(s)
I20260929 21:04:28.156533 20290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-0/data/info.pb
I20260929 21:04:28.163353 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 20290
I20260929 21:04:28.163516 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-0/wal/instance
I20260929 21:04:28.163955 388 external_mini_cluster.cc:1428] Setting key 5b68aa52c2da6f71dfb46bf0443264ac
I20260929 21:04:28.166555 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36891
--builtin_ntp_servers=127.0.97.20:39079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 21:04:28.169059 20419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:28.159023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43855 (local address 127.0.97.62:36891)
0929 21:04:28.159507 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:28.159512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:28.160491 (+ 979us) server_negotiation.cc:407] Connection header received
0929 21:04:28.162345 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:28.162351 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:28.162440 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:28.162594 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:28.163723 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:28.164484 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:28.166254 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:28.166711 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:28.166826 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:28.168195 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:28.168199 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:28.168203 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:28.168221 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:28.168283 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:28.168299 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:28.168401 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:28.168676 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 21:04:28.168752 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":271,"threads_started":1}
I20260929 21:04:28.170465 20420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:28.159460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36891 (local address 127.0.97.1:43855)
0929 21:04:28.160333 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:28.160374 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:28.162068 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:28.162831 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:28.162839 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:28.162924 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:04:28.163527 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:28.163546 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:28.164635 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:28.164640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:28.166099 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:28.166111 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:28.166922 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:28.166931 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:28.167158 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:28.167880 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 21:04:28.167883 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:04:28.167920 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 21:04:28.167993 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:04:28.168425 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:04:28.168440 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:04:28.168452 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 21:04:28.168651 (+ 199us) client_negotiation.cc:240] Negotiation successful
0929 21:04:28.169086 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":283,"threads_started":1}
I20260929 21:04:28.172019 20417 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36891
I20260929 21:04:28.172327 20417 heartbeater.cc:464] Registering TS with master...
I20260929 21:04:28.173113 20417 heartbeater.cc:511] Master 127.0.97.62:36891 requested a full tablet report, sending...
I20260929 21:04:28.175446 20231 ts_manager.cc:194] Registered new tserver with Master: 564de47e367f46a3810ca2934a71db6e (127.0.97.1:37733)
I20260929 21:04:28.177763 20231 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:43855
W20260929 21:04:28.490760 20421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:28.491022 20421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:28.491107 20421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:28.498284 20421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:28.498437 20421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:28.498596 20421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 21:04:28.509158 20421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:28.510448 20421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:28.512243 20421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:28.525077 20426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:28.525130 20429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:28.525152 20427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:28.525992 20421 server_base.cc:1034] running on GCE node
I20260929 21:04:28.526638 20421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:28.527657 20421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:28.528983 20421 hybrid_clock.cc:648] HybridClock initialized: now 1790715868528922 us; error 50 us; skew 500 ppm
I20260929 21:04:28.529335 20421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 21:04:28.531574 20421 webserver.cc:460] Webserver started at http://127.0.97.2:33439/ using document root <none> and password file <none>
I20260929 21:04:28.532358 20421 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:28.532488 20421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:28.532812 20421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:28.535662 20421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-1/data/instance:
uuid: "cecbd245b1c940b0b3d50d8947ec45e4"
format_stamp: "Formatted at 2026-09-29 21:04:28 on dist-test-slave-1dms"
server_key: "df6ced145e5ce32141260e7f4ecae550"
server_key_iv: "14b6c60813c993f2f6ab5b1eb3848018"
server_key_version: "encryptionkey@0"
I20260929 21:04:28.536463 20421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-1/wal/instance:
uuid: "cecbd245b1c940b0b3d50d8947ec45e4"
format_stamp: "Formatted at 2026-09-29 21:04:28 on dist-test-slave-1dms"
server_key: "df6ced145e5ce32141260e7f4ecae550"
server_key_iv: "14b6c60813c993f2f6ab5b1eb3848018"
server_key_version: "encryptionkey@0"
I20260929 21:04:28.541428 20421 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260929 21:04:28.544790 20435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:28.546111 20421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 21:04:28.546293 20421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-1/wal
uuid: "cecbd245b1c940b0b3d50d8947ec45e4"
format_stamp: "Formatted at 2026-09-29 21:04:28 on dist-test-slave-1dms"
server_key: "df6ced145e5ce32141260e7f4ecae550"
server_key_iv: "14b6c60813c993f2f6ab5b1eb3848018"
server_key_version: "encryptionkey@0"
I20260929 21:04:28.546525 20421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:28.572573 20421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:28.573501 20421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:28.573794 20421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:28.574790 20421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:04:28.576327 20421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:04:28.576448 20421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:28.576604 20421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:04:28.576705 20421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:28.616333 20421 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46109
I20260929 21:04:28.616420 20547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46109 every 8 connection(s)
I20260929 21:04:28.617703 20421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-1/data/info.pb
I20260929 21:04:28.623925 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 20421
I20260929 21:04:28.624114 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-1/wal/instance
I20260929 21:04:28.624545 388 external_mini_cluster.cc:1428] Setting key f546c73e7476c90b6b0c245564e0cf7a
I20260929 21:04:28.627295 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36891
--builtin_ntp_servers=127.0.97.20:39079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 21:04:28.630899 20419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:28.620128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45581 (local address 127.0.97.62:36891)
0929 21:04:28.620306 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:28.620311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:28.621831 (+ 1520us) server_negotiation.cc:407] Connection header received
0929 21:04:28.623601 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:28.623610 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:28.623707 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:28.623832 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:28.625029 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:28.625891 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:28.627723 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:28.628478 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:28.628629 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:28.630007 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:28.630015 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:28.630023 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:28.630047 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:28.630088 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:28.630105 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:28.630199 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:28.630408 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 21:04:28.630516 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 21:04:28.631771 20550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:28.620683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36891 (local address 127.0.97.2:45581)
0929 21:04:28.621652 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:28.621703 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:28.623335 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:28.624028 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:28.624040 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:28.624150 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:04:28.624812 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:28.624869 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:28.626051 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:28.626056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:28.627575 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:28.627587 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:28.628702 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:28.628706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:28.628965 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:28.629683 (+ 718us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 21:04:28.629686 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:04:28.629726 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 21:04:28.629810 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:04:28.630201 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:04:28.630208 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:04:28.630214 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 21:04:28.630349 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 21:04:28.630648 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":314,"threads_started":1}
I20260929 21:04:28.633255 20548 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36891
I20260929 21:04:28.633579 20548 heartbeater.cc:464] Registering TS with master...
I20260929 21:04:28.634469 20548 heartbeater.cc:511] Master 127.0.97.62:36891 requested a full tablet report, sending...
I20260929 21:04:28.636121 20231 ts_manager.cc:194] Registered new tserver with Master: cecbd245b1c940b0b3d50d8947ec45e4 (127.0.97.2:46109)
I20260929 21:04:28.637249 20231 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:45581
W20260929 21:04:28.951901 20551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:28.952263 20551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:28.952436 20551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:28.959604 20551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:28.959704 20551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:28.959880 20551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 21:04:28.968961 20551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:28.970222 20551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:28.971944 20551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:28.984961 20557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:28.984913 20556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:28.985008 20559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:28.985729 20551 server_base.cc:1034] running on GCE node
I20260929 21:04:28.986320 20551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:28.987322 20551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:28.988516 20551 hybrid_clock.cc:648] HybridClock initialized: now 1790715868988472 us; error 35 us; skew 500 ppm
I20260929 21:04:28.988875 20551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 21:04:28.991004 20551 webserver.cc:460] Webserver started at http://127.0.97.3:36655/ using document root <none> and password file <none>
I20260929 21:04:28.991683 20551 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:28.991798 20551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:28.992117 20551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:28.994933 20551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-2/data/instance:
uuid: "3c58d6be0fac4dd09e025bd035b7b2ff"
format_stamp: "Formatted at 2026-09-29 21:04:28 on dist-test-slave-1dms"
server_key: "5057635edbbed21811601cfeebfe5dec"
server_key_iv: "61c02992118681be9e32d1d7dcfbb325"
server_key_version: "encryptionkey@0"
I20260929 21:04:28.995700 20551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-2/wal/instance:
uuid: "3c58d6be0fac4dd09e025bd035b7b2ff"
format_stamp: "Formatted at 2026-09-29 21:04:28 on dist-test-slave-1dms"
server_key: "5057635edbbed21811601cfeebfe5dec"
server_key_iv: "61c02992118681be9e32d1d7dcfbb325"
server_key_version: "encryptionkey@0"
I20260929 21:04:29.000788 20551 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 21:04:29.004014 20565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:29.005391 20551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:04:29.005545 20551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-2/wal
uuid: "3c58d6be0fac4dd09e025bd035b7b2ff"
format_stamp: "Formatted at 2026-09-29 21:04:28 on dist-test-slave-1dms"
server_key: "5057635edbbed21811601cfeebfe5dec"
server_key_iv: "61c02992118681be9e32d1d7dcfbb325"
server_key_version: "encryptionkey@0"
I20260929 21:04:29.005807 20551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:29.022809 20551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:29.023589 20551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:29.023867 20551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:29.024788 20551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 21:04:29.026201 20551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 21:04:29.026288 20551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:29.026439 20551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 21:04:29.026541 20551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:29.059079 20551 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42559
I20260929 21:04:29.059129 20677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42559 every 8 connection(s)
I20260929 21:04:29.060397 20551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-2/data/info.pb
I20260929 21:04:29.063926 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 20551
I20260929 21:04:29.064100 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-2/wal/instance
I20260929 21:04:29.064532 388 external_mini_cluster.cc:1428] Setting key 7a7d4974f194f8323b4a36d4c1d477c6
I20260929 21:04:29.073202 20419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:29.063149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49293 (local address 127.0.97.62:36891)
0929 21:04:29.063333 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:29.063338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:29.064831 (+ 1493us) server_negotiation.cc:407] Connection header received
0929 21:04:29.066907 (+ 2076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:29.066914 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:29.066998 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:29.067111 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:29.068190 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:29.068986 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:29.070509 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:29.070938 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:29.071069 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:29.072385 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:29.072390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:29.072393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:29.072411 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:29.072442 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:29.072454 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:29.072548 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:29.072760 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 21:04:29.072880 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 21:04:29.074105 20680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:29.063582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36891 (local address 127.0.97.3:49293)
0929 21:04:29.064659 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:29.064704 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:29.066661 (+ 1957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:29.067378 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:29.067386 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:29.067472 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:04:29.068016 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:29.068032 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:29.069126 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:29.069132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:29.070364 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:29.070375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:29.071152 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:29.071169 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:29.071383 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:29.072070 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 21:04:29.072072 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:04:29.072109 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 21:04:29.072192 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:04:29.072546 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:04:29.072558 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:04:29.072567 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 21:04:29.072750 (+ 183us) client_negotiation.cc:240] Negotiation successful
0929 21:04:29.073137 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":367,"threads_started":1}
I20260929 21:04:29.075309 20678 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36891
I20260929 21:04:29.075553 20678 heartbeater.cc:464] Registering TS with master...
I20260929 21:04:29.076190 20678 heartbeater.cc:511] Master 127.0.97.62:36891 requested a full tablet report, sending...
I20260929 21:04:29.077489 20231 ts_manager.cc:194] Registered new tserver with Master: 3c58d6be0fac4dd09e025bd035b7b2ff (127.0.97.3:42559)
I20260929 21:04:29.078474 20231 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:49293
I20260929 21:04:29.079571 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 21:04:29.089860 20419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:29.083939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50448 (local address 127.0.97.62:36891)
0929 21:04:29.084116 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:29.084123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:29.084528 (+ 405us) server_negotiation.cc:407] Connection header received
0929 21:04:29.084726 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:29.084732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:29.084814 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:29.084948 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:29.085421 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:29.086099 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:29.087564 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:29.088027 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:29.088146 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:29.089148 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:29.089154 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:29.089158 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:29.089195 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:29.089226 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:29.089237 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:29.089329 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:29.089469 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 21:04:29.089555 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 21:04:29.093346 20231 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50448:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 21:04:29.100278 20231 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 21:04:29.138278 20693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:29.112225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34746 (local address 127.0.97.3:42559)
0929 21:04:29.129763 (+ 17538us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:29.129771 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:29.129803 (+ 32us) server_negotiation.cc:407] Connection header received
0929 21:04:29.129884 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:29.129890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:29.130100 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:29.130324 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:04:29.131018 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:29.132172 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:29.135493 (+ 3321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:29.137025 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:29.137159 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:29.137460 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:29.137727 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 21:04:29.137905 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17344,"thread_start_us":17177,"threads_started":1}
I20260929 21:04:29.138278 20690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:29.111923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42559 (local address 127.0.0.1:34746)
0929 21:04:29.127206 (+ 15283us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:29.127225 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:29.127342 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:29.130340 (+ 2998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:29.130349 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:29.130393 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:04:29.130804 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:29.130821 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:29.132382 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:29.132392 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:29.135269 (+ 2877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:29.135286 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:29.137231 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:29.137237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:29.137439 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:29.137502 (+ 63us) client_negotiation.cc:769] Sending connection context
0929 21:04:29.137673 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 21:04:29.137894 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15122,"thread_start_us":13678,"threads_started":1}
I20260929 21:04:29.141664 20691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:29.111681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46109 (local address 127.0.0.1:46378)
0929 21:04:29.125037 (+ 13356us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:29.125107 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:29.125354 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:29.132686 (+ 7332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:29.132692 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:29.132721 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:04:29.133194 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:29.133214 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:29.135683 (+ 2469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:29.135692 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:29.138226 (+ 2534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:29.138245 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:29.140669 (+ 2424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:29.140676 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:29.140945 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:29.140985 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 21:04:29.141104 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 21:04:29.141249 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12931,"mutex_wait_us":6,"thread_start_us":14875,"threads_started":1}
I20260929 21:04:29.141696 20692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:29.111985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46378 (local address 127.0.97.2:46109)
0929 21:04:29.127915 (+ 15930us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:29.127929 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:29.127978 (+ 49us) server_negotiation.cc:407] Connection header received
0929 21:04:29.128166 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:29.128180 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:29.128552 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:29.128871 (+ 319us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:04:29.133933 (+ 5062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:29.135545 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:29.138464 (+ 2919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:29.140079 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:29.140227 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:29.140521 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:29.141190 (+ 669us) server_negotiation.cc:299] Negotiation successful
0929 21:04:29.141368 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15673,"thread_start_us":18593,"threads_started":1}
I20260929 21:04:29.142932 20613 tablet_service.cc:1467] Processing CreateTablet for tablet 613ac99c89d342768cb61e9ac4140fca (DEFAULT_TABLE table=test-table [id=6be23c3047cf42c7954be1bf4775448b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:04:29.144450 20613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 613ac99c89d342768cb61e9ac4140fca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:04:29.145467 20483 tablet_service.cc:1467] Processing CreateTablet for tablet 613ac99c89d342768cb61e9ac4140fca (DEFAULT_TABLE table=test-table [id=6be23c3047cf42c7954be1bf4775448b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:04:29.146634 20483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 613ac99c89d342768cb61e9ac4140fca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:04:29.149865 20695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:29.127004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37733 (local address 127.0.0.1:49598)
0929 21:04:29.136193 (+ 9189us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:29.136222 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:29.136463 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:29.137495 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:29.137500 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:29.137526 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:04:29.137953 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:29.137964 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:29.140186 (+ 2222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:29.140192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:29.142455 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:29.142468 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:29.149003 (+ 6535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:29.149009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:29.149209 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:29.149249 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 21:04:29.149349 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 21:04:29.149488 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9029,"mutex_wait_us":462,"thread_start_us":8401,"threads_started":1}
I20260929 21:04:29.150278 20694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:29.127199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49598 (local address 127.0.97.1:37733)
0929 21:04:29.133791 (+ 6592us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:29.133802 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:29.136332 (+ 2530us) server_negotiation.cc:407] Connection header received
0929 21:04:29.136707 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:29.136717 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:29.137039 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:29.137227 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:04:29.138190 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:29.139980 (+ 1790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:29.142659 (+ 2679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:29.148760 (+ 6101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:29.148957 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:29.149217 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:29.149422 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 21:04:29.149687 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6429,"thread_start_us":6276,"threads_started":1}
I20260929 21:04:29.154109 20352 tablet_service.cc:1467] Processing CreateTablet for tablet 613ac99c89d342768cb61e9ac4140fca (DEFAULT_TABLE table=test-table [id=6be23c3047cf42c7954be1bf4775448b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 21:04:29.155684 20352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 613ac99c89d342768cb61e9ac4140fca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:04:29.162915 20697 tablet_bootstrap.cc:492] T 613ac99c89d342768cb61e9ac4140fca P cecbd245b1c940b0b3d50d8947ec45e4: Bootstrap starting.
I20260929 21:04:29.163996 20696 tablet_bootstrap.cc:492] T 613ac99c89d342768cb61e9ac4140fca P 3c58d6be0fac4dd09e025bd035b7b2ff: Bootstrap starting.
I20260929 21:04:29.166628 20697 tablet_bootstrap.cc:654] T 613ac99c89d342768cb61e9ac4140fca P cecbd245b1c940b0b3d50d8947ec45e4: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:29.167721 20697 log.cc:824] T 613ac99c89d342768cb61e9ac4140fca P cecbd245b1c940b0b3d50d8947ec45e4: Log is configured to *not* fsync() on all Append() calls
I20260929 21:04:29.168164 20696 tablet_bootstrap.cc:654] T 613ac99c89d342768cb61e9ac4140fca P 3c58d6be0fac4dd09e025bd035b7b2ff: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:29.169467 20696 log.cc:824] T 613ac99c89d342768cb61e9ac4140fca P 3c58d6be0fac4dd09e025bd035b7b2ff: Log is configured to *not* fsync() on all Append() calls
I20260929 21:04:29.171977 20697 tablet_bootstrap.cc:492] T 613ac99c89d342768cb61e9ac4140fca P cecbd245b1c940b0b3d50d8947ec45e4: No bootstrap required, opened a new log
I20260929 21:04:29.172323 20697 ts_tablet_manager.cc:1397] T 613ac99c89d342768cb61e9ac4140fca P cecbd245b1c940b0b3d50d8947ec45e4: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260929 21:04:29.174817 20696 tablet_bootstrap.cc:492] T 613ac99c89d342768cb61e9ac4140fca P 3c58d6be0fac4dd09e025bd035b7b2ff: No bootstrap required, opened a new log
I20260929 21:04:29.175074 20696 ts_tablet_manager.cc:1397] T 613ac99c89d342768cb61e9ac4140fca P 3c58d6be0fac4dd09e025bd035b7b2ff: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260929 21:04:29.181483 20417 heartbeater.cc:503] Master 127.0.97.62:36891 was elected leader, sending a full tablet report...
I20260929 21:04:29.181777 20700 tablet_bootstrap.cc:492] T 613ac99c89d342768cb61e9ac4140fca P 564de47e367f46a3810ca2934a71db6e: Bootstrap starting.
I20260929 21:04:29.181479 20697 raft_consensus.cc:348] T 613ac99c89d342768cb61e9ac4140fca P cecbd245b1c940b0b3d50d8947ec45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cecbd245b1c940b0b3d50d8947ec45e4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "3c58d6be0fac4dd09e025bd035b7b2ff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42559 } } peers { permanent_uuid: "564de47e367f46a3810ca2934a71db6e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37733 } }
I20260929 21:04:29.182003 20697 raft_consensus.cc:374] T 613ac99c89d342768cb61e9ac4140fca P cecbd245b1c940b0b3d50d8947ec45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:29.182124 20697 raft_consensus.cc:729] T 613ac99c89d342768cb61e9ac4140fca P cecbd245b1c940b0b3d50d8947ec45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cecbd245b1c940b0b3d50d8947ec45e4, State: Initialized, Role: FOLLOWER
I20260929 21:04:29.182855 20697 consensus_queue.cc:260] T 613ac99c89d342768cb61e9ac4140fca P cecbd245b1c940b0b3d50d8947ec45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cecbd245b1c940b0b3d50d8947ec45e4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "3c58d6be0fac4dd09e025bd035b7b2ff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42559 } } peers { permanent_uuid: "564de47e367f46a3810ca2934a71db6e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37733 } }
I20260929 21:04:29.184798 20697 ts_tablet_manager.cc:1428] T 613ac99c89d342768cb61e9ac4140fca P cecbd245b1c940b0b3d50d8947ec45e4: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260929 21:04:29.185272 20548 heartbeater.cc:503] Master 127.0.97.62:36891 was elected leader, sending a full tablet report...
I20260929 21:04:29.187403 20696 raft_consensus.cc:348] T 613ac99c89d342768cb61e9ac4140fca P 3c58d6be0fac4dd09e025bd035b7b2ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cecbd245b1c940b0b3d50d8947ec45e4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "3c58d6be0fac4dd09e025bd035b7b2ff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42559 } } peers { permanent_uuid: "564de47e367f46a3810ca2934a71db6e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37733 } }
I20260929 21:04:29.187862 20696 raft_consensus.cc:374] T 613ac99c89d342768cb61e9ac4140fca P 3c58d6be0fac4dd09e025bd035b7b2ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:29.187969 20696 raft_consensus.cc:729] T 613ac99c89d342768cb61e9ac4140fca P 3c58d6be0fac4dd09e025bd035b7b2ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c58d6be0fac4dd09e025bd035b7b2ff, State: Initialized, Role: FOLLOWER
I20260929 21:04:29.188687 20700 tablet_bootstrap.cc:654] T 613ac99c89d342768cb61e9ac4140fca P 564de47e367f46a3810ca2934a71db6e: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:29.188691 20696 consensus_queue.cc:260] T 613ac99c89d342768cb61e9ac4140fca P 3c58d6be0fac4dd09e025bd035b7b2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cecbd245b1c940b0b3d50d8947ec45e4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "3c58d6be0fac4dd09e025bd035b7b2ff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42559 } } peers { permanent_uuid: "564de47e367f46a3810ca2934a71db6e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37733 } }
I20260929 21:04:29.189860 20678 heartbeater.cc:503] Master 127.0.97.62:36891 was elected leader, sending a full tablet report...
I20260929 21:04:29.189980 20700 log.cc:824] T 613ac99c89d342768cb61e9ac4140fca P 564de47e367f46a3810ca2934a71db6e: Log is configured to *not* fsync() on all Append() calls
I20260929 21:04:29.190338 20696 ts_tablet_manager.cc:1428] T 613ac99c89d342768cb61e9ac4140fca P 3c58d6be0fac4dd09e025bd035b7b2ff: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260929 21:04:29.192792 20700 tablet_bootstrap.cc:492] T 613ac99c89d342768cb61e9ac4140fca P 564de47e367f46a3810ca2934a71db6e: No bootstrap required, opened a new log
I20260929 21:04:29.193140 20700 ts_tablet_manager.cc:1397] T 613ac99c89d342768cb61e9ac4140fca P 564de47e367f46a3810ca2934a71db6e: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.002s
I20260929 21:04:29.200685 20700 raft_consensus.cc:348] T 613ac99c89d342768cb61e9ac4140fca P 564de47e367f46a3810ca2934a71db6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cecbd245b1c940b0b3d50d8947ec45e4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "3c58d6be0fac4dd09e025bd035b7b2ff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42559 } } peers { permanent_uuid: "564de47e367f46a3810ca2934a71db6e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37733 } }
I20260929 21:04:29.201157 20700 raft_consensus.cc:374] T 613ac99c89d342768cb61e9ac4140fca P 564de47e367f46a3810ca2934a71db6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:29.201274 20700 raft_consensus.cc:729] T 613ac99c89d342768cb61e9ac4140fca P 564de47e367f46a3810ca2934a71db6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 564de47e367f46a3810ca2934a71db6e, State: Initialized, Role: FOLLOWER
I20260929 21:04:29.201932 20700 consensus_queue.cc:260] T 613ac99c89d342768cb61e9ac4140fca P 564de47e367f46a3810ca2934a71db6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cecbd245b1c940b0b3d50d8947ec45e4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "3c58d6be0fac4dd09e025bd035b7b2ff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42559 } } peers { permanent_uuid: "564de47e367f46a3810ca2934a71db6e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37733 } }
I20260929 21:04:29.202046 20701 raft_consensus.cc:482] T 613ac99c89d342768cb61e9ac4140fca P cecbd245b1c940b0b3d50d8947ec45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 21:04:29.202363 20701 raft_consensus.cc:504] T 613ac99c89d342768cb61e9ac4140fca P cecbd245b1c940b0b3d50d8947ec45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cecbd245b1c940b0b3d50d8947ec45e4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "3c58d6be0fac4dd09e025bd035b7b2ff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42559 } } peers { permanent_uuid: "564de47e367f46a3810ca2934a71db6e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37733 } }
I20260929 21:04:29.203766 20700 ts_tablet_manager.cc:1428] T 613ac99c89d342768cb61e9ac4140fca P 564de47e367f46a3810ca2934a71db6e: Time spent starting tablet: real 0.011s user 0.002s sys 0.007s
I20260929 21:04:29.204232 20701 leader_election.cc:288] T 613ac99c89d342768cb61e9ac4140fca P cecbd245b1c940b0b3d50d8947ec45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c58d6be0fac4dd09e025bd035b7b2ff (127.0.97.3:42559), 564de47e367f46a3810ca2934a71db6e (127.0.97.1:37733)
I20260929 21:04:29.212338 20694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:29.205050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52127 (local address 127.0.97.1:37733)
0929 21:04:29.205259 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:29.205265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:29.206287 (+ 1022us) server_negotiation.cc:407] Connection header received
0929 21:04:29.206546 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:29.206552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:29.206639 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:29.206749 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:04:29.207459 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:29.208205 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:29.210654 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:29.211376 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:29.211510 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:29.211613 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:29.211992 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 21:04:29.212109 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 21:04:29.212399 20705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:29.204588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42559 (local address 127.0.97.2:50337)
0929 21:04:29.205492 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:29.205509 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:29.205696 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:29.207030 (+ 1334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:29.207033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:29.207066 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:04:29.207478 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:29.207488 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:29.208571 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:29.208574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:29.210192 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:29.210204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:29.211623 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:29.211631 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:29.211796 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:29.211820 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 21:04:29.211894 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 21:04:29.212035 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"mutex_wait_us":7,"thread_start_us":550,"threads_started":1}
I20260929 21:04:29.212499 20706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:29.204602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37733 (local address 127.0.97.2:52127)
0929 21:04:29.206186 (+ 1584us) negotiation.cc:107] Waiting for socket to connect
0929 21:04:29.206208 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 21:04:29.206383 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:04:29.206907 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:04:29.206914 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:04:29.206942 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 21:04:29.207290 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:29.207300 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:29.208340 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:29.208345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:29.210468 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:04:29.210486 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:04:29.211609 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:04:29.211616 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:04:29.211810 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:29.211835 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 21:04:29.211954 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 21:04:29.212096 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1464,"mutex_wait_us":762,"thread_start_us":509,"threads_started":1}
I20260929 21:04:29.212359 20693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:29.205101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50337 (local address 127.0.97.3:42559)
0929 21:04:29.205274 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:29.205280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:29.205600 (+ 320us) server_negotiation.cc:407] Connection header received
0929 21:04:29.206619 (+ 1019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:29.206625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:29.206715 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:29.206786 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 21:04:29.207605 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:29.208427 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:29.210390 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:29.211376 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:29.211558 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:29.211663 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:29.211955 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 21:04:29.212090 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 21:04:29.213693 20633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "613ac99c89d342768cb61e9ac4140fca" candidate_uuid: "cecbd245b1c940b0b3d50d8947ec45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c58d6be0fac4dd09e025bd035b7b2ff" is_pre_election: true
I20260929 21:04:29.213698 20372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "613ac99c89d342768cb61e9ac4140fca" candidate_uuid: "cecbd245b1c940b0b3d50d8947ec45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "564de47e367f46a3810ca2934a71db6e" is_pre_election: true
I20260929 21:04:29.214165 20372 raft_consensus.cc:2454] T 613ac99c89d342768cb61e9ac4140fca P 564de47e367f46a3810ca2934a71db6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cecbd245b1c940b0b3d50d8947ec45e4 in term 0.
I20260929 21:04:29.214170 20633 raft_consensus.cc:2454] T 613ac99c89d342768cb61e9ac4140fca P 3c58d6be0fac4dd09e025bd035b7b2ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cecbd245b1c940b0b3d50d8947ec45e4 in term 0.
I20260929 21:04:29.214813 20439 leader_election.cc:302] T 613ac99c89d342768cb61e9ac4140fca P cecbd245b1c940b0b3d50d8947ec45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c58d6be0fac4dd09e025bd035b7b2ff, cecbd245b1c940b0b3d50d8947ec45e4; no voters:
I20260929 21:04:29.215224 20701 raft_consensus.cc:2781] T 613ac99c89d342768cb61e9ac4140fca P cecbd245b1c940b0b3d50d8947ec45e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 21:04:29.215389 20701 raft_consensus.cc:482] T 613ac99c89d342768cb61e9ac4140fca P cecbd245b1c940b0b3d50d8947ec45e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 21:04:29.215503 20701 raft_consensus.cc:3037] T 613ac99c89d342768cb61e9ac4140fca P cecbd245b1c940b0b3d50d8947ec45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:29.218156 20701 raft_consensus.cc:504] T 613ac99c89d342768cb61e9ac4140fca P cecbd245b1c940b0b3d50d8947ec45e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cecbd245b1c940b0b3d50d8947ec45e4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "3c58d6be0fac4dd09e025bd035b7b2ff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42559 } } peers { permanent_uuid: "564de47e367f46a3810ca2934a71db6e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37733 } }
I20260929 21:04:29.218825 20701 leader_election.cc:288] T 613ac99c89d342768cb61e9ac4140fca P cecbd245b1c940b0b3d50d8947ec45e4 [CANDIDATE]: Term 1 election: Requested vote from peers 3c58d6be0fac4dd09e025bd035b7b2ff (127.0.97.3:42559), 564de47e367f46a3810ca2934a71db6e (127.0.97.1:37733)
I20260929 21:04:29.219116 20633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "613ac99c89d342768cb61e9ac4140fca" candidate_uuid: "cecbd245b1c940b0b3d50d8947ec45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c58d6be0fac4dd09e025bd035b7b2ff"
I20260929 21:04:29.219321 20633 raft_consensus.cc:3037] T 613ac99c89d342768cb61e9ac4140fca P 3c58d6be0fac4dd09e025bd035b7b2ff [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:29.219318 20372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "613ac99c89d342768cb61e9ac4140fca" candidate_uuid: "cecbd245b1c940b0b3d50d8947ec45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "564de47e367f46a3810ca2934a71db6e"
I20260929 21:04:29.219525 20372 raft_consensus.cc:3037] T 613ac99c89d342768cb61e9ac4140fca P 564de47e367f46a3810ca2934a71db6e [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:29.221598 20633 raft_consensus.cc:2454] T 613ac99c89d342768cb61e9ac4140fca P 3c58d6be0fac4dd09e025bd035b7b2ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cecbd245b1c940b0b3d50d8947ec45e4 in term 1.
I20260929 21:04:29.221966 20439 leader_election.cc:302] T 613ac99c89d342768cb61e9ac4140fca P cecbd245b1c940b0b3d50d8947ec45e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c58d6be0fac4dd09e025bd035b7b2ff, cecbd245b1c940b0b3d50d8947ec45e4; no voters:
I20260929 21:04:29.222074 20372 raft_consensus.cc:2454] T 613ac99c89d342768cb61e9ac4140fca P 564de47e367f46a3810ca2934a71db6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cecbd245b1c940b0b3d50d8947ec45e4 in term 1.
I20260929 21:04:29.222247 20701 raft_consensus.cc:2781] T 613ac99c89d342768cb61e9ac4140fca P cecbd245b1c940b0b3d50d8947ec45e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:04:29.222734 20701 raft_consensus.cc:686] T 613ac99c89d342768cb61e9ac4140fca P cecbd245b1c940b0b3d50d8947ec45e4 [term 1 LEADER]: Becoming Leader. State: Replica: cecbd245b1c940b0b3d50d8947ec45e4, State: Running, Role: LEADER
I20260929 21:04:29.223191 20701 consensus_queue.cc:237] T 613ac99c89d342768cb61e9ac4140fca P cecbd245b1c940b0b3d50d8947ec45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cecbd245b1c940b0b3d50d8947ec45e4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "3c58d6be0fac4dd09e025bd035b7b2ff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42559 } } peers { permanent_uuid: "564de47e367f46a3810ca2934a71db6e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37733 } }
I20260929 21:04:29.227921 20231 catalog_manager.cc:5495] T 613ac99c89d342768cb61e9ac4140fca P cecbd245b1c940b0b3d50d8947ec45e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to cecbd245b1c940b0b3d50d8947ec45e4 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "cecbd245b1c940b0b3d50d8947ec45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cecbd245b1c940b0b3d50d8947ec45e4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c58d6be0fac4dd09e025bd035b7b2ff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "564de47e367f46a3810ca2934a71db6e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37733 } health_report { overall_health: UNKNOWN } } }
I20260929 21:04:29.249162 20692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:29.243300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46384 (local address 127.0.97.2:46109)
0929 21:04:29.243494 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:29.243500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:29.243549 (+ 49us) server_negotiation.cc:407] Connection header received
0929 21:04:29.243781 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:29.243785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:29.243884 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:29.244060 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:04:29.244443 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:29.245275 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:29.246856 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:29.247318 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:29.247457 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:29.247984 (+ 527us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:04:29.248495 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:04:29.248666 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:29.248732 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 21:04:29.248813 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 21:04:29.257370 20633 raft_consensus.cc:1261] T 613ac99c89d342768cb61e9ac4140fca P 3c58d6be0fac4dd09e025bd035b7b2ff [term 1 FOLLOWER]: Refusing update from remote peer cecbd245b1c940b0b3d50d8947ec45e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:04:29.257376 20372 raft_consensus.cc:1261] T 613ac99c89d342768cb61e9ac4140fca P 564de47e367f46a3810ca2934a71db6e [term 1 FOLLOWER]: Refusing update from remote peer cecbd245b1c940b0b3d50d8947ec45e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:04:29.258369 20707 consensus_queue.cc:1035] T 613ac99c89d342768cb61e9ac4140fca P cecbd245b1c940b0b3d50d8947ec45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "564de47e367f46a3810ca2934a71db6e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:04:29.258714 20701 consensus_queue.cc:1035] T 613ac99c89d342768cb61e9ac4140fca P cecbd245b1c940b0b3d50d8947ec45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c58d6be0fac4dd09e025bd035b7b2ff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:04:29.269897 20710 mvcc.cc:205] Tried to move back new op lower bound from 7334772200468066304 to 7334772200341991424. Current Snapshot: MvccSnapshot[applied={T|T < 7334772200468066304}]
I20260929 21:04:29.286413 20230 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 21:04:29.286783 20230 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 21:04:29.290194 20230 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d39758ce551745409923e8db9555586b: Sending DeleteTablet for 3 replicas of tablet 613ac99c89d342768cb61e9ac4140fca
I20260929 21:04:29.291337 20483 tablet_service.cc:1514] Processing DeleteTablet for tablet 613ac99c89d342768cb61e9ac4140fca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:04:29 UTC) from {username='slave'} at 127.0.0.1:46378
I20260929 21:04:29.291541 20613 tablet_service.cc:1514] Processing DeleteTablet for tablet 613ac99c89d342768cb61e9ac4140fca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:04:29 UTC) from {username='slave'} at 127.0.0.1:34746
I20260929 21:04:29.292246 20352 tablet_service.cc:1514] Processing DeleteTablet for tablet 613ac99c89d342768cb61e9ac4140fca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 21:04:29 UTC) from {username='slave'} at 127.0.0.1:49598
I20260929 21:04:29.293216 20722 tablet_replica.cc:330] T 613ac99c89d342768cb61e9ac4140fca P 564de47e367f46a3810ca2934a71db6e: stopping tablet replica
I20260929 21:04:29.293531 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 20290
I20260929 21:04:29.293694 20722 raft_consensus.cc:2229] T 613ac99c89d342768cb61e9ac4140fca P 564de47e367f46a3810ca2934a71db6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:04:29.294548 20722 raft_consensus.cc:2258] T 613ac99c89d342768cb61e9ac4140fca P 564de47e367f46a3810ca2934a71db6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:04:29.297669 20722 ts_tablet_manager.cc:1905] T 613ac99c89d342768cb61e9ac4140fca P 564de47e367f46a3810ca2934a71db6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:04:29.305367 20722 ts_tablet_manager.cc:1918] T 613ac99c89d342768cb61e9ac4140fca P 564de47e367f46a3810ca2934a71db6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:04:29.305613 20722 log.cc:1196] T 613ac99c89d342768cb61e9ac4140fca P 564de47e367f46a3810ca2934a71db6e: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-0/wal/wals/613ac99c89d342768cb61e9ac4140fca
I20260929 21:04:29.306330 20722 ts_tablet_manager.cc:1939] T 613ac99c89d342768cb61e9ac4140fca P 564de47e367f46a3810ca2934a71db6e: Deleting consensus metadata
I20260929 21:04:29.308461 20216 catalog_manager.cc:4841] TS 564de47e367f46a3810ca2934a71db6e (127.0.97.1:37733): tablet 613ac99c89d342768cb61e9ac4140fca (table test-table [id=6be23c3047cf42c7954be1bf4775448b]) successfully deleted
I20260929 21:04:29.313836 20721 tablet_replica.cc:330] T 613ac99c89d342768cb61e9ac4140fca P 3c58d6be0fac4dd09e025bd035b7b2ff: stopping tablet replica
I20260929 21:04:29.314280 20721 raft_consensus.cc:2229] T 613ac99c89d342768cb61e9ac4140fca P 3c58d6be0fac4dd09e025bd035b7b2ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 21:04:29.314772 20721 raft_consensus.cc:2258] T 613ac99c89d342768cb61e9ac4140fca P 3c58d6be0fac4dd09e025bd035b7b2ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:04:29.317138 20720 tablet_replica.cc:330] T 613ac99c89d342768cb61e9ac4140fca P cecbd245b1c940b0b3d50d8947ec45e4: stopping tablet replica
I20260929 21:04:29.317564 20720 raft_consensus.cc:2229] T 613ac99c89d342768cb61e9ac4140fca P cecbd245b1c940b0b3d50d8947ec45e4 [term 1 LEADER]: Raft consensus shutting down.
I20260929 21:04:29.318208 20720 raft_consensus.cc:2258] T 613ac99c89d342768cb61e9ac4140fca P cecbd245b1c940b0b3d50d8947ec45e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 21:04:29.319790 20721 ts_tablet_manager.cc:1905] T 613ac99c89d342768cb61e9ac4140fca P 3c58d6be0fac4dd09e025bd035b7b2ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:04:29.320150 20694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:29.296042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49606 (local address 127.0.97.1:37733)
0929 21:04:29.296238 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:29.296244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:29.309386 (+ 13142us) server_negotiation.cc:407] Connection header received
0929 21:04:29.309674 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:29.309679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:29.309789 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:29.309862 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:29.310528 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:29.311665 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:29.316602 (+ 4937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:29.317276 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:29.317418 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:29.318771 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:29.318778 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:29.318790 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:29.319324 (+ 534us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:29.319366 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:29.319383 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:29.319507 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:29.319653 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 21:04:29.319773 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 21:04:29.321329 20412 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:04:29.322270 20720 ts_tablet_manager.cc:1905] T 613ac99c89d342768cb61e9ac4140fca P cecbd245b1c940b0b3d50d8947ec45e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 21:04:29.326615 20721 ts_tablet_manager.cc:1918] T 613ac99c89d342768cb61e9ac4140fca P 3c58d6be0fac4dd09e025bd035b7b2ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:04:29.326777 20721 log.cc:1196] T 613ac99c89d342768cb61e9ac4140fca P 3c58d6be0fac4dd09e025bd035b7b2ff: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-2/wal/wals/613ac99c89d342768cb61e9ac4140fca
I20260929 21:04:29.327291 20721 ts_tablet_manager.cc:1939] T 613ac99c89d342768cb61e9ac4140fca P 3c58d6be0fac4dd09e025bd035b7b2ff: Deleting consensus metadata
I20260929 21:04:29.329028 20218 catalog_manager.cc:4841] TS 3c58d6be0fac4dd09e025bd035b7b2ff (127.0.97.3:42559): tablet 613ac99c89d342768cb61e9ac4140fca (table test-table [id=6be23c3047cf42c7954be1bf4775448b]) successfully deleted
I20260929 21:04:29.332472 20720 ts_tablet_manager.cc:1918] T 613ac99c89d342768cb61e9ac4140fca P cecbd245b1c940b0b3d50d8947ec45e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 21:04:29.332697 20720 log.cc:1196] T 613ac99c89d342768cb61e9ac4140fca P cecbd245b1c940b0b3d50d8947ec45e4: Deleting WAL directory at /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790715556572573-388-0/minicluster-data/ts-1/wal/wals/613ac99c89d342768cb61e9ac4140fca
I20260929 21:04:29.333480 20720 ts_tablet_manager.cc:1939] T 613ac99c89d342768cb61e9ac4140fca P cecbd245b1c940b0b3d50d8947ec45e4: Deleting consensus metadata
I20260929 21:04:29.335424 20216 catalog_manager.cc:4841] TS cecbd245b1c940b0b3d50d8947ec45e4 (127.0.97.2:46109): tablet 613ac99c89d342768cb61e9ac4140fca (table test-table [id=6be23c3047cf42c7954be1bf4775448b]) successfully deleted
I20260929 21:04:29.420989 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 20290
I20260929 21:04:29.445526 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 20421
I20260929 21:04:29.452646 20692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:29.446302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46390 (local address 127.0.97.2:46109)
0929 21:04:29.446482 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:29.446503 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:29.446525 (+ 22us) server_negotiation.cc:407] Connection header received
0929 21:04:29.446741 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:29.446751 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:29.446828 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:29.446944 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:29.447357 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:29.448102 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:29.449468 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:29.449946 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:29.450100 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:29.451240 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:29.451249 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:29.451264 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:29.451874 (+ 610us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:29.451923 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:29.451942 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:29.452020 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:29.452199 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 21:04:29.452276 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 21:04:29.453419 20543 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:04:29.563846 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 20421
I20260929 21:04:29.582343 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 20551
I20260929 21:04:29.589313 20693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:29.583062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34758 (local address 127.0.97.3:42559)
0929 21:04:29.583260 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:29.583265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:29.583290 (+ 25us) server_negotiation.cc:407] Connection header received
0929 21:04:29.583497 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:29.583509 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:29.583589 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:29.583694 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:29.584113 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:29.584975 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:29.586243 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:29.586651 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:29.586782 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:29.587944 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:29.587953 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:29.587969 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:29.588542 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:29.588589 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:29.588608 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:29.588737 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:29.588899 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 21:04:29.589009 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 21:04:29.589957 20673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:04:29.678921 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 20551
I20260929 21:04:29.696317 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 20200
I20260929 21:04:29.697464 20261 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:04:29.810407 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 20200
2026-09-29T21:04:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2662 ms)
[----------] 6 tests from EncryptionPolicyTest (20708 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 21:04:29 dist-test-slave-1dms krb5kdc[20730](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 21:04:29 dist-test-slave-1dms krb5kdc[20730](info): set up 2 sockets
Sep 29 21:04:29 dist-test-slave-1dms krb5kdc[20730](info): commencing operation
krb5kdc: starting...
W20260929 21:04:30.917505 388 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 21:04:31 dist-test-slave-1dms krb5kdc[20730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T21:04:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:04:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 21:04:31.168622 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43363
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:04:31.491221 20743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:31.491489 20743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:31.491683 20743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:31.498721 20743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:04:31.498831 20743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:31.498910 20743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:31.499012 20743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:04:31.499071 20743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:04:31.509276 20743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43363
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:31.510593 20743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:31.511827 20743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:31.523195 20751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:31.523305 20748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:31.523254 20749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:31.524058 20743 server_base.cc:1034] running on GCE node
I20260929 21:04:31.524905 20743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:31.526319 20743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:31.527558 20743 hybrid_clock.cc:648] HybridClock initialized: now 1790715871527514 us; error 65 us; skew 500 ppm
I20260929 21:04:31.527933 20743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:04:31 dist-test-slave-1dms krb5kdc[20730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:04:31.531241 20743 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 21:04:31.532902 20743 webserver.cc:460] Webserver started at http://127.0.0.1:46753/ using document root <none> and password file <none>
I20260929 21:04:31.533634 20743 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:31.533790 20743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:31.534129 20743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:31.537333 20743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "2e54f7483c6f4dc6bc480e6ebfe9486c"
format_stamp: "Formatted at 2026-09-29 21:04:31 on dist-test-slave-1dms"
server_key: "933e1d6d7a8ba951e07674e19a7f6487"
server_key_iv: "4c12f98043be8a69459805774a452e33"
server_key_version: "encryptionkey@0"
I20260929 21:04:31.538136 20743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "2e54f7483c6f4dc6bc480e6ebfe9486c"
format_stamp: "Formatted at 2026-09-29 21:04:31 on dist-test-slave-1dms"
server_key: "933e1d6d7a8ba951e07674e19a7f6487"
server_key_iv: "4c12f98043be8a69459805774a452e33"
server_key_version: "encryptionkey@0"
I20260929 21:04:31.543462 20743 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 21:04:31.547192 20758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:31.548609 20743 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 21:04:31.548926 20743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "2e54f7483c6f4dc6bc480e6ebfe9486c"
format_stamp: "Formatted at 2026-09-29 21:04:31 on dist-test-slave-1dms"
server_key: "933e1d6d7a8ba951e07674e19a7f6487"
server_key_iv: "4c12f98043be8a69459805774a452e33"
server_key_version: "encryptionkey@0"
I20260929 21:04:31.549136 20743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:31.561911 20743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:31.566640 20743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:31.566951 20743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:31.585970 20743 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43363
I20260929 21:04:31.585994 20809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43363 every 8 connection(s)
I20260929 21:04:31.587404 20743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:04:31.591202 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 20743
I20260929 21:04:31.591418 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:04:31.591410 20810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:04:31.592105 388 external_mini_cluster.cc:1428] Setting key b914374750a1837bca5c5ecbb0554ead
I20260929 21:04:31.606681 20810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e54f7483c6f4dc6bc480e6ebfe9486c: Bootstrap starting.
Sep 29 21:04:31 dist-test-slave-1dms krb5kdc[20730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 21:04:31.610324 20810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e54f7483c6f4dc6bc480e6ebfe9486c: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:31.611390 20810 log.cc:824] T 00000000000000000000000000000000 P 2e54f7483c6f4dc6bc480e6ebfe9486c: Log is configured to *not* fsync() on all Append() calls
I20260929 21:04:31.614146 20810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e54f7483c6f4dc6bc480e6ebfe9486c: No bootstrap required, opened a new log
I20260929 21:04:31.615720 20813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:31.594121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43497 (local address 127.0.0.1:43363)
0929 21:04:31.595008 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:31.595023 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:31.595072 (+ 49us) server_negotiation.cc:407] Connection header received
0929 21:04:31.596754 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:31.596788 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:31.597346 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:31.597799 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:31.599072 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:31.600992 (+ 1920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:31.602308 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:31.603399 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:31.603588 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:31.608362 (+ 4774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:31.608374 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:31.608401 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:04:31.608455 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:31.611725 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:31.612508 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:31.612515 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:31.612520 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:31.612686 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:31.613179 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:31.613183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:31.613185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:31.613674 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:31.613989 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:31.614467 (+ 478us) server_negotiation.cc:299] Negotiation successful
0929 21:04:31.614788 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":367,"threads_started":1}
I20260929 21:04:31.619786 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 21:04:31.622345 20810 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e54f7483c6f4dc6bc480e6ebfe9486c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e54f7483c6f4dc6bc480e6ebfe9486c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43363 } }
I20260929 21:04:31.622807 20810 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e54f7483c6f4dc6bc480e6ebfe9486c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:31.622929 20810 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e54f7483c6f4dc6bc480e6ebfe9486c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e54f7483c6f4dc6bc480e6ebfe9486c, State: Initialized, Role: FOLLOWER
I20260929 21:04:31.623608 20810 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e54f7483c6f4dc6bc480e6ebfe9486c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e54f7483c6f4dc6bc480e6ebfe9486c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43363 } }
I20260929 21:04:31.623854 20810 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e54f7483c6f4dc6bc480e6ebfe9486c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:04:31.624017 20810 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e54f7483c6f4dc6bc480e6ebfe9486c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:04:31.624218 20810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e54f7483c6f4dc6bc480e6ebfe9486c [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:31.626921 20810 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e54f7483c6f4dc6bc480e6ebfe9486c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e54f7483c6f4dc6bc480e6ebfe9486c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43363 } }
I20260929 21:04:31.627451 20810 leader_election.cc:302] T 00000000000000000000000000000000 P 2e54f7483c6f4dc6bc480e6ebfe9486c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e54f7483c6f4dc6bc480e6ebfe9486c; no voters:
I20260929 21:04:31.628122 20810 leader_election.cc:288] T 00000000000000000000000000000000 P 2e54f7483c6f4dc6bc480e6ebfe9486c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 21:04:31.628242 20820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e54f7483c6f4dc6bc480e6ebfe9486c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:04:31.630817 20810 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e54f7483c6f4dc6bc480e6ebfe9486c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:04:31.634859 20820 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e54f7483c6f4dc6bc480e6ebfe9486c [term 1 LEADER]: Becoming Leader. State: Replica: 2e54f7483c6f4dc6bc480e6ebfe9486c, State: Running, Role: LEADER
I20260929 21:04:31.635639 20820 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e54f7483c6f4dc6bc480e6ebfe9486c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e54f7483c6f4dc6bc480e6ebfe9486c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43363 } }
I20260929 21:04:31.638875 20821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e54f7483c6f4dc6bc480e6ebfe9486c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e54f7483c6f4dc6bc480e6ebfe9486c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e54f7483c6f4dc6bc480e6ebfe9486c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43363 } } }
I20260929 21:04:31.639246 20821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e54f7483c6f4dc6bc480e6ebfe9486c [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:31.639664 20813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:31.624544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54439 (local address 127.0.0.1:43363)
0929 21:04:31.624740 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:31.624749 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:31.625176 (+ 427us) server_negotiation.cc:407] Connection header received
0929 21:04:31.625382 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:31.625387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:31.625459 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:31.625581 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:31.626949 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:31.627854 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:31.629662 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:31.630108 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:31.630258 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:31.632895 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:31.632904 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:31.632910 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:04:31.632962 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:31.636058 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:31.636940 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:31.636949 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:31.636952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:31.637066 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:31.637761 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:31.637765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:31.637768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:31.638281 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:31.638610 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:31.639034 (+ 424us) server_negotiation.cc:299] Negotiation successful
0929 21:04:31.639251 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 21:04:31.639815 20822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e54f7483c6f4dc6bc480e6ebfe9486c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e54f7483c6f4dc6bc480e6ebfe9486c. Latest consensus state: current_term: 1 leader_uuid: "2e54f7483c6f4dc6bc480e6ebfe9486c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e54f7483c6f4dc6bc480e6ebfe9486c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43363 } } }
I20260929 21:04:31.640902 20822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e54f7483c6f4dc6bc480e6ebfe9486c [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:31.645994 20830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:04:31.650689 20830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:04:31.658598 20830 catalog_manager.cc:1347] Generated new cluster ID: 103c0f221d62488e8c3cba5d51e6fd2f
I20260929 21:04:31.658730 20830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:04:31.678869 20830 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:04:31.680482 20830 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:04:31.691551 20830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e54f7483c6f4dc6bc480e6ebfe9486c: Generated new TSK 0
I20260929 21:04:31.692420 20830 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 21:04:31.707671 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 20743
I20260929 21:04:31.709189 20805 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:04:31.803483 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 20743
W20260929 21:04:31.821023 20728 connection.cc:535] client connection to 127.0.0.1:43363 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43363): Connection reset by peer (error 104)
2026-09-29T21:04:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1988 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 21:04:31 dist-test-slave-1dms krb5kdc[20842](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 21:04:31 dist-test-slave-1dms krb5kdc[20842](info): set up 2 sockets
Sep 29 21:04:31 dist-test-slave-1dms krb5kdc[20842](info): commencing operation
krb5kdc: starting...
W20260929 21:04:32.901319 388 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 21:04:33 dist-test-slave-1dms krb5kdc[20842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T21:04:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:04:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 21:04:33.142402 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44795
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:04:33.470367 20855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:33.470661 20855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:33.470860 20855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:33.478279 20855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:04:33.478392 20855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:33.478497 20855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:33.478605 20855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:04:33.478690 20855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:04:33.488570 20855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44795
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:33.489867 20855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:33.491153 20855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:33.503283 20860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:33.503227 20861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:33.503170 20863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:33.503937 20855 server_base.cc:1034] running on GCE node
I20260929 21:04:33.504660 20855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:33.506039 20855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:33.507238 20855 hybrid_clock.cc:648] HybridClock initialized: now 1790715873507192 us; error 38 us; skew 500 ppm
I20260929 21:04:33.507573 20855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:04:33 dist-test-slave-1dms krb5kdc[20842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:04:33.510722 20855 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 21:04:33.512080 20855 webserver.cc:460] Webserver started at http://127.0.0.1:34729/ using document root <none> and password file <none>
I20260929 21:04:33.512743 20855 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:33.512903 20855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:33.513211 20855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:33.516041 20855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "0f0fe4d8b24045198f9876ed4caa5f9c"
format_stamp: "Formatted at 2026-09-29 21:04:33 on dist-test-slave-1dms"
server_key: "be405de47de49d895f30d763b439a30b"
server_key_iv: "2385329f4b1940ee83922168125bdf28"
server_key_version: "encryptionkey@0"
I20260929 21:04:33.516791 20855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "0f0fe4d8b24045198f9876ed4caa5f9c"
format_stamp: "Formatted at 2026-09-29 21:04:33 on dist-test-slave-1dms"
server_key: "be405de47de49d895f30d763b439a30b"
server_key_iv: "2385329f4b1940ee83922168125bdf28"
server_key_version: "encryptionkey@0"
I20260929 21:04:33.521770 20855 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260929 21:04:33.525089 20870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:33.526376 20855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 21:04:33.526576 20855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "0f0fe4d8b24045198f9876ed4caa5f9c"
format_stamp: "Formatted at 2026-09-29 21:04:33 on dist-test-slave-1dms"
server_key: "be405de47de49d895f30d763b439a30b"
server_key_iv: "2385329f4b1940ee83922168125bdf28"
server_key_version: "encryptionkey@0"
I20260929 21:04:33.526790 20855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:33.547210 20855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:33.551704 20855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:33.551994 20855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:33.570598 20855 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44795
I20260929 21:04:33.570631 20921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44795 every 8 connection(s)
I20260929 21:04:33.572003 20855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:04:33.575659 20922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:04:33.577275 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 20855
I20260929 21:04:33.577448 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:04:33.577975 388 external_mini_cluster.cc:1428] Setting key 946a77ce57ceb7a3751afd499e138921
I20260929 21:04:33.588165 20922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f0fe4d8b24045198f9876ed4caa5f9c: Bootstrap starting.
I20260929 21:04:33.590480 20925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:33.579774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44901 (local address 127.0.0.1:44795)
0929 21:04:33.580541 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:33.580556 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:33.580594 (+ 38us) server_negotiation.cc:407] Connection header received
0929 21:04:33.582247 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:33.582268 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:33.582657 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:33.583045 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:33.583552 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:33.584789 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:33.586136 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:33.586824 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:33.587012 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:33.588272 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:33.588284 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:33.588303 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:33.588894 (+ 591us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:33.588982 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:33.589006 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:33.589163 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:33.589434 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 21:04:33.589601 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":368,"threads_started":1}
I20260929 21:04:33.591410 20922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f0fe4d8b24045198f9876ed4caa5f9c: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:33.592705 20922 log.cc:824] T 00000000000000000000000000000000 P 0f0fe4d8b24045198f9876ed4caa5f9c: Log is configured to *not* fsync() on all Append() calls
I20260929 21:04:33.594414 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 21:04:33.595777 20922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f0fe4d8b24045198f9876ed4caa5f9c: No bootstrap required, opened a new log
I20260929 21:04:33.605885 20925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:33.599397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60375 (local address 127.0.0.1:44795)
0929 21:04:33.599568 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:33.599574 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:33.599899 (+ 325us) server_negotiation.cc:407] Connection header received
0929 21:04:33.600037 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:33.600042 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:33.600117 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:33.600233 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:33.600723 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:33.601933 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:33.603219 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:33.603926 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:33.604062 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:33.605107 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:33.605117 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:33.605125 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:33.605148 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:33.605215 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:33.605234 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:33.605345 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:33.605509 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 21:04:33.605566 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 21:04:33.605577 20922 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f0fe4d8b24045198f9876ed4caa5f9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f0fe4d8b24045198f9876ed4caa5f9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44795 } }
I20260929 21:04:33.606030 20922 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f0fe4d8b24045198f9876ed4caa5f9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:33.606158 20922 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f0fe4d8b24045198f9876ed4caa5f9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f0fe4d8b24045198f9876ed4caa5f9c, State: Initialized, Role: FOLLOWER
I20260929 21:04:33.606839 20922 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f0fe4d8b24045198f9876ed4caa5f9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f0fe4d8b24045198f9876ed4caa5f9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44795 } }
I20260929 21:04:33.607115 20922 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f0fe4d8b24045198f9876ed4caa5f9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:04:33.607288 20922 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f0fe4d8b24045198f9876ed4caa5f9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:04:33.607465 20922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f0fe4d8b24045198f9876ed4caa5f9c [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:33.609786 20922 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f0fe4d8b24045198f9876ed4caa5f9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f0fe4d8b24045198f9876ed4caa5f9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44795 } }
I20260929 21:04:33.610260 20922 leader_election.cc:302] T 00000000000000000000000000000000 P 0f0fe4d8b24045198f9876ed4caa5f9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f0fe4d8b24045198f9876ed4caa5f9c; no voters:
I20260929 21:04:33.610824 20922 leader_election.cc:288] T 00000000000000000000000000000000 P 0f0fe4d8b24045198f9876ed4caa5f9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 21:04:33.610962 20932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f0fe4d8b24045198f9876ed4caa5f9c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:04:33.612252 20932 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f0fe4d8b24045198f9876ed4caa5f9c [term 1 LEADER]: Becoming Leader. State: Replica: 0f0fe4d8b24045198f9876ed4caa5f9c, State: Running, Role: LEADER
I20260929 21:04:33.612776 20932 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f0fe4d8b24045198f9876ed4caa5f9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f0fe4d8b24045198f9876ed4caa5f9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44795 } }
I20260929 21:04:33.613164 20922 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f0fe4d8b24045198f9876ed4caa5f9c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:04:33.615916 20934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f0fe4d8b24045198f9876ed4caa5f9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f0fe4d8b24045198f9876ed4caa5f9c. Latest consensus state: current_term: 1 leader_uuid: "0f0fe4d8b24045198f9876ed4caa5f9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f0fe4d8b24045198f9876ed4caa5f9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44795 } } }
I20260929 21:04:33.615784 20933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f0fe4d8b24045198f9876ed4caa5f9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f0fe4d8b24045198f9876ed4caa5f9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f0fe4d8b24045198f9876ed4caa5f9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44795 } } }
I20260929 21:04:33.617547 20934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f0fe4d8b24045198f9876ed4caa5f9c [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:33.617556 20933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f0fe4d8b24045198f9876ed4caa5f9c [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:33.619567 20944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:04:33.624038 20944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:04:33.632373 20944 catalog_manager.cc:1347] Generated new cluster ID: 0cd5c215ebe3444282f04c3dc55003b5
I20260929 21:04:33.632488 20944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:04:33.646042 20944 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:04:33.647389 20944 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:04:33.655634 20944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f0fe4d8b24045198f9876ed4caa5f9c: Generated new TSK 0
I20260929 21:04:33.656687 20944 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 21:04:33.664110 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 20855
W20260929 21:04:33.665715 20917 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44901
W20260929 21:04:33.666534 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 20855: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 21:04:33.666752 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 20855
W20260929 21:04:33.685539 20840 connection.cc:535] client connection to 127.0.0.1:44795 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44795): Connection reset by peer (error 104)
2026-09-29T21:04:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1864 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 21:04:33 dist-test-slave-1dms krb5kdc[20953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 21:04:33 dist-test-slave-1dms krb5kdc[20953](info): set up 2 sockets
Sep 29 21:04:33 dist-test-slave-1dms krb5kdc[20953](info): commencing operation
krb5kdc: starting...
W20260929 21:04:34.769270 388 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 21:04:34 dist-test-slave-1dms krb5kdc[20953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T21:04:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:04:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 21:04:35.016347 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39473
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:04:35.338362 20966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:35.338649 20966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:35.338841 20966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:35.346053 20966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:04:35.346161 20966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:35.346268 20966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:35.346375 20966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:04:35.346460 20966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:04:35.356144 20966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39473
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:35.357460 20966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:35.358759 20966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:35.370551 20971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:35.370575 20972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:35.370615 20974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:35.371435 20966 server_base.cc:1034] running on GCE node
I20260929 21:04:35.372169 20966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:35.373587 20966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:35.374794 20966 hybrid_clock.cc:648] HybridClock initialized: now 1790715875374757 us; error 42 us; skew 500 ppm
I20260929 21:04:35.375134 20966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:04:35 dist-test-slave-1dms krb5kdc[20953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:04:35.391379 20966 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 21:04:35.393086 20966 webserver.cc:460] Webserver started at http://127.0.0.1:40047/ using document root <none> and password file <none>
I20260929 21:04:35.393810 20966 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:35.393960 20966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:35.394289 20966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:35.397361 20966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "da950a515e0b426d9373b70444b1215b"
format_stamp: "Formatted at 2026-09-29 21:04:35 on dist-test-slave-1dms"
server_key: "3ef7a3855ec11da20f396f32cc045b56"
server_key_iv: "8e159c131552315ed6580fd4c41b6945"
server_key_version: "encryptionkey@0"
I20260929 21:04:35.398124 20966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "da950a515e0b426d9373b70444b1215b"
format_stamp: "Formatted at 2026-09-29 21:04:35 on dist-test-slave-1dms"
server_key: "3ef7a3855ec11da20f396f32cc045b56"
server_key_iv: "8e159c131552315ed6580fd4c41b6945"
server_key_version: "encryptionkey@0"
I20260929 21:04:35.403326 20966 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 21:04:35.406816 20981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:35.408119 20966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 21:04:35.408305 20966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "da950a515e0b426d9373b70444b1215b"
format_stamp: "Formatted at 2026-09-29 21:04:35 on dist-test-slave-1dms"
server_key: "3ef7a3855ec11da20f396f32cc045b56"
server_key_iv: "8e159c131552315ed6580fd4c41b6945"
server_key_version: "encryptionkey@0"
I20260929 21:04:35.408505 20966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:35.414574 20966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:35.419519 20966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:35.419828 20966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:35.438462 20966 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39473
I20260929 21:04:35.438549 21032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39473 every 8 connection(s)
I20260929 21:04:35.439922 20966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:04:35.443939 21033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:04:35.449348 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 20966
I20260929 21:04:35.449575 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:04:35.450155 388 external_mini_cluster.cc:1428] Setting key 14dd89af74eb378825134518e62e717c
I20260929 21:04:35.456414 21033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da950a515e0b426d9373b70444b1215b: Bootstrap starting.
I20260929 21:04:35.458190 21036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:35.452092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44173 (local address 127.0.0.1:39473)
0929 21:04:35.452943 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:35.452954 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:35.452999 (+ 45us) server_negotiation.cc:407] Connection header received
0929 21:04:35.454764 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:35.454787 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:35.455127 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:35.455522 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:35.456570 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:35.456580 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:35.456598 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:35.456617 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:35.456690 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:35.456714 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:35.456967 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:35.457106 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 21:04:35.457294 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":343,"threads_started":1}
I20260929 21:04:35.460469 21033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da950a515e0b426d9373b70444b1215b: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:35.461961 21033 log.cc:824] T 00000000000000000000000000000000 P da950a515e0b426d9373b70444b1215b: Log is configured to *not* fsync() on all Append() calls
I20260929 21:04:35.462522 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 21:04:35.464819 21033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da950a515e0b426d9373b70444b1215b: No bootstrap required, opened a new log
I20260929 21:04:35.469434 21036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:35.467036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37617 (local address 127.0.0.1:39473)
0929 21:04:35.467241 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:35.467247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:35.467469 (+ 222us) server_negotiation.cc:407] Connection header received
0929 21:04:35.467626 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:35.467636 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:35.467702 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:35.467810 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:35.468678 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:35.468684 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:35.468688 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:35.468706 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:35.468757 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:35.468778 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:35.468930 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:35.469093 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 21:04:35.469154 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 21:04:35.473786 21033 raft_consensus.cc:348] T 00000000000000000000000000000000 P da950a515e0b426d9373b70444b1215b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da950a515e0b426d9373b70444b1215b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39473 } }
I20260929 21:04:35.474150 21033 raft_consensus.cc:374] T 00000000000000000000000000000000 P da950a515e0b426d9373b70444b1215b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:35.474273 21033 raft_consensus.cc:729] T 00000000000000000000000000000000 P da950a515e0b426d9373b70444b1215b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da950a515e0b426d9373b70444b1215b, State: Initialized, Role: FOLLOWER
I20260929 21:04:35.474828 21033 consensus_queue.cc:260] T 00000000000000000000000000000000 P da950a515e0b426d9373b70444b1215b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da950a515e0b426d9373b70444b1215b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39473 } }
I20260929 21:04:35.475059 21033 raft_consensus.cc:388] T 00000000000000000000000000000000 P da950a515e0b426d9373b70444b1215b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:04:35.475203 21033 raft_consensus.cc:482] T 00000000000000000000000000000000 P da950a515e0b426d9373b70444b1215b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:04:35.475353 21033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da950a515e0b426d9373b70444b1215b [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:35.477818 21033 raft_consensus.cc:504] T 00000000000000000000000000000000 P da950a515e0b426d9373b70444b1215b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da950a515e0b426d9373b70444b1215b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39473 } }
I20260929 21:04:35.478319 21033 leader_election.cc:302] T 00000000000000000000000000000000 P da950a515e0b426d9373b70444b1215b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da950a515e0b426d9373b70444b1215b; no voters:
I20260929 21:04:35.478899 21033 leader_election.cc:288] T 00000000000000000000000000000000 P da950a515e0b426d9373b70444b1215b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 21:04:35.479061 21043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da950a515e0b426d9373b70444b1215b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:04:35.479732 21043 raft_consensus.cc:686] T 00000000000000000000000000000000 P da950a515e0b426d9373b70444b1215b [term 1 LEADER]: Becoming Leader. State: Replica: da950a515e0b426d9373b70444b1215b, State: Running, Role: LEADER
I20260929 21:04:35.480269 21043 consensus_queue.cc:237] T 00000000000000000000000000000000 P da950a515e0b426d9373b70444b1215b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da950a515e0b426d9373b70444b1215b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39473 } }
I20260929 21:04:35.481000 21033 sys_catalog.cc:564] T 00000000000000000000000000000000 P da950a515e0b426d9373b70444b1215b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:04:35.482937 21044 sys_catalog.cc:455] T 00000000000000000000000000000000 P da950a515e0b426d9373b70444b1215b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da950a515e0b426d9373b70444b1215b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da950a515e0b426d9373b70444b1215b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39473 } } }
I20260929 21:04:35.483323 21044 sys_catalog.cc:458] T 00000000000000000000000000000000 P da950a515e0b426d9373b70444b1215b [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:35.485225 21045 sys_catalog.cc:455] T 00000000000000000000000000000000 P da950a515e0b426d9373b70444b1215b [sys.catalog]: SysCatalogTable state changed. Reason: New leader da950a515e0b426d9373b70444b1215b. Latest consensus state: current_term: 1 leader_uuid: "da950a515e0b426d9373b70444b1215b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da950a515e0b426d9373b70444b1215b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39473 } } }
I20260929 21:04:35.485576 21045 sys_catalog.cc:458] T 00000000000000000000000000000000 P da950a515e0b426d9373b70444b1215b [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:35.486194 21049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:04:35.491449 21049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:04:35.499961 21049 catalog_manager.cc:1347] Generated new cluster ID: f713b04dd3a7463b9d12c9d65b5333a9
I20260929 21:04:35.500072 21049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:04:35.517419 21049 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:04:35.517925 21049 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:04:35.532931 21049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da950a515e0b426d9373b70444b1215b: Generated new TSK 0
I20260929 21:04:35.533776 21049 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 21:04:35.541723 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 20966
W20260929 21:04:35.542795 21028 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44173
W20260929 21:04:35.543484 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 20966: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 21:04:35.543633 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 20966
2026-09-29T21:04:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1874 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 21:04:35 dist-test-slave-1dms krb5kdc[21064](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 21:04:35 dist-test-slave-1dms krb5kdc[21064](info): set up 2 sockets
Sep 29 21:04:35 dist-test-slave-1dms krb5kdc[21064](info): commencing operation
krb5kdc: starting...
W20260929 21:04:36.633287 388 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 21:04:36 dist-test-slave-1dms krb5kdc[21064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T21:04:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:04:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 21:04:36.883961 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39243
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:04:37.211066 21077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:37.211436 21077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:37.211611 21077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:37.218917 21077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:04:37.219067 21077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:37.219187 21077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:37.219291 21077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:04:37.219374 21077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:04:37.229048 21077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:37.230333 21077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:37.231673 21077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:37.243398 21083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:37.243736 21085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:37.243986 21082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:37.244189 21077 server_base.cc:1034] running on GCE node
I20260929 21:04:37.244992 21077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:37.246320 21077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:37.247547 21077 hybrid_clock.cc:648] HybridClock initialized: now 1790715877247498 us; error 40 us; skew 500 ppm
I20260929 21:04:37.247893 21077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:04:37 dist-test-slave-1dms krb5kdc[21064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:04:37.250933 21077 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 21:04:37.252281 21077 webserver.cc:460] Webserver started at http://127.0.0.1:43373/ using document root <none> and password file <none>
I20260929 21:04:37.253010 21077 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:37.253152 21077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:37.253451 21077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:37.256258 21077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "c00529afe0594497b607b6c64a9acd86"
format_stamp: "Formatted at 2026-09-29 21:04:37 on dist-test-slave-1dms"
server_key: "6ee101513bd2e244bfd64f1c65dc6bad"
server_key_iv: "ec693f37825b376d28c11d9eeae04230"
server_key_version: "encryptionkey@0"
I20260929 21:04:37.257061 21077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "c00529afe0594497b607b6c64a9acd86"
format_stamp: "Formatted at 2026-09-29 21:04:37 on dist-test-slave-1dms"
server_key: "6ee101513bd2e244bfd64f1c65dc6bad"
server_key_iv: "ec693f37825b376d28c11d9eeae04230"
server_key_version: "encryptionkey@0"
I20260929 21:04:37.261938 21077 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260929 21:04:37.265350 21092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:37.266676 21077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 21:04:37.266857 21077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "c00529afe0594497b607b6c64a9acd86"
format_stamp: "Formatted at 2026-09-29 21:04:37 on dist-test-slave-1dms"
server_key: "6ee101513bd2e244bfd64f1c65dc6bad"
server_key_iv: "ec693f37825b376d28c11d9eeae04230"
server_key_version: "encryptionkey@0"
I20260929 21:04:37.267051 21077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:37.282236 21077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:37.285969 21077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:37.286247 21077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:37.304381 21077 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39243
I20260929 21:04:37.304400 21143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39243 every 8 connection(s)
I20260929 21:04:37.305711 21077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:04:37.307628 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 21077
I20260929 21:04:37.307811 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:04:37.308357 388 external_mini_cluster.cc:1428] Setting key 44cb2b7b11f8c86e95fc65364ff64187
I20260929 21:04:37.310272 21144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 21:04:37 dist-test-slave-1dms krb5kdc[21064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 21:04:37.322283 21144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c00529afe0594497b607b6c64a9acd86: Bootstrap starting.
I20260929 21:04:37.325567 21144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c00529afe0594497b607b6c64a9acd86: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:37.326551 21144 log.cc:824] T 00000000000000000000000000000000 P c00529afe0594497b607b6c64a9acd86: Log is configured to *not* fsync() on all Append() calls
I20260929 21:04:37.329313 21144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c00529afe0594497b607b6c64a9acd86: No bootstrap required, opened a new log
I20260929 21:04:37.330374 21147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:37.310105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51405 (local address 127.0.0.1:39243)
0929 21:04:37.310927 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:37.310938 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:37.310977 (+ 39us) server_negotiation.cc:407] Connection header received
0929 21:04:37.312540 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:37.312560 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:37.312992 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:37.313320 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:37.315087 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:37.316386 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:37.318054 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:37.318741 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:37.318964 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:37.323354 (+ 4390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:37.323361 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:37.323374 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:04:37.323409 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:37.326439 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:37.327145 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:37.327152 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:37.327157 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:37.327341 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:37.327723 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:37.327730 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:37.327733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:37.328208 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:37.328516 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:37.328976 (+ 460us) server_negotiation.cc:299] Negotiation successful
0929 21:04:37.329420 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":402,"threads_started":1}
I20260929 21:04:37.333931 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 21:04:37.338688 21144 raft_consensus.cc:348] T 00000000000000000000000000000000 P c00529afe0594497b607b6c64a9acd86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c00529afe0594497b607b6c64a9acd86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39243 } }
I20260929 21:04:37.339151 21144 raft_consensus.cc:374] T 00000000000000000000000000000000 P c00529afe0594497b607b6c64a9acd86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:37.339326 21144 raft_consensus.cc:729] T 00000000000000000000000000000000 P c00529afe0594497b607b6c64a9acd86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c00529afe0594497b607b6c64a9acd86, State: Initialized, Role: FOLLOWER
I20260929 21:04:37.340008 21144 consensus_queue.cc:260] T 00000000000000000000000000000000 P c00529afe0594497b607b6c64a9acd86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c00529afe0594497b607b6c64a9acd86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39243 } }
I20260929 21:04:37.340304 21144 raft_consensus.cc:388] T 00000000000000000000000000000000 P c00529afe0594497b607b6c64a9acd86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:04:37.340440 21144 raft_consensus.cc:482] T 00000000000000000000000000000000 P c00529afe0594497b607b6c64a9acd86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:04:37.340626 21144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c00529afe0594497b607b6c64a9acd86 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:37.343587 21144 raft_consensus.cc:504] T 00000000000000000000000000000000 P c00529afe0594497b607b6c64a9acd86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c00529afe0594497b607b6c64a9acd86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39243 } }
I20260929 21:04:37.344143 21144 leader_election.cc:302] T 00000000000000000000000000000000 P c00529afe0594497b607b6c64a9acd86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c00529afe0594497b607b6c64a9acd86; no voters:
I20260929 21:04:37.344847 21144 leader_election.cc:288] T 00000000000000000000000000000000 P c00529afe0594497b607b6c64a9acd86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 21:04:37.344990 21154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c00529afe0594497b607b6c64a9acd86 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:04:37.345887 21154 raft_consensus.cc:686] T 00000000000000000000000000000000 P c00529afe0594497b607b6c64a9acd86 [term 1 LEADER]: Becoming Leader. State: Replica: c00529afe0594497b607b6c64a9acd86, State: Running, Role: LEADER
I20260929 21:04:37.346550 21154 consensus_queue.cc:237] T 00000000000000000000000000000000 P c00529afe0594497b607b6c64a9acd86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c00529afe0594497b607b6c64a9acd86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39243 } }
I20260929 21:04:37.347379 21144 sys_catalog.cc:564] T 00000000000000000000000000000000 P c00529afe0594497b607b6c64a9acd86 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:04:37.349789 21155 sys_catalog.cc:455] T 00000000000000000000000000000000 P c00529afe0594497b607b6c64a9acd86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c00529afe0594497b607b6c64a9acd86. Latest consensus state: current_term: 1 leader_uuid: "c00529afe0594497b607b6c64a9acd86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c00529afe0594497b607b6c64a9acd86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39243 } } }
I20260929 21:04:37.350246 21155 sys_catalog.cc:458] T 00000000000000000000000000000000 P c00529afe0594497b607b6c64a9acd86 [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:37.351228 21156 sys_catalog.cc:455] T 00000000000000000000000000000000 P c00529afe0594497b607b6c64a9acd86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c00529afe0594497b607b6c64a9acd86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c00529afe0594497b607b6c64a9acd86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39243 } } }
I20260929 21:04:37.351660 21156 sys_catalog.cc:458] T 00000000000000000000000000000000 P c00529afe0594497b607b6c64a9acd86 [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:37.353598 21161 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:04:37.355785 21147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:37.339065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58999 (local address 127.0.0.1:39243)
0929 21:04:37.339284 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:37.339290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:37.339626 (+ 336us) server_negotiation.cc:407] Connection header received
0929 21:04:37.339787 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:37.339791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:37.339847 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:37.339922 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:37.341836 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:37.342565 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:37.344335 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:37.344903 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:37.345053 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:37.347662 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:37.347668 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:37.347674 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:04:37.347717 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:37.352797 (+ 5080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:37.353730 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:37.353736 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:37.353740 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:37.353900 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:37.354272 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:37.354278 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:37.354281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:37.354575 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:37.354856 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:37.355156 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 21:04:37.355387 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 21:04:37.359844 21161 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:04:37.371920 21161 catalog_manager.cc:1347] Generated new cluster ID: d16842a643e14dde93f0b446ab4b68a7
I20260929 21:04:37.372035 21161 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:04:37.420341 21161 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:04:37.422058 21161 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:04:37.430114 21161 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c00529afe0594497b607b6c64a9acd86: Generated new TSK 0
I20260929 21:04:37.430961 21161 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 21:04:37.442900 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 21077
I20260929 21:04:37.444453 21139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:04:37.539712 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 21077
2026-09-29T21:04:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1996 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 21:04:37 dist-test-slave-1dms krb5kdc[21176](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 21:04:37 dist-test-slave-1dms krb5kdc[21176](info): set up 2 sockets
Sep 29 21:04:37 dist-test-slave-1dms krb5kdc[21176](info): commencing operation
krb5kdc: starting...
W20260929 21:04:38.637279 388 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 21:04:38 dist-test-slave-1dms krb5kdc[21176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T21:04:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:04:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 21:04:38.876593 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40053
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:04:39.205561 21189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:39.205842 21189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:39.206032 21189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:39.213222 21189 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:04:39.213321 21189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:39.213441 21189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:39.213539 21189 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:04:39.213629 21189 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:04:39.223315 21189 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:39.224558 21189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:39.225878 21189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:39.237329 21194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:39.237325 21195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:39.237450 21197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:39.238116 21189 server_base.cc:1034] running on GCE node
I20260929 21:04:39.238875 21189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:39.240240 21189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:39.241694 21189 hybrid_clock.cc:648] HybridClock initialized: now 1790715879241667 us; error 37 us; skew 500 ppm
I20260929 21:04:39.242069 21189 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:04:39 dist-test-slave-1dms krb5kdc[21176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:04:39.245095 21189 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 21:04:39.246460 21189 webserver.cc:460] Webserver started at http://127.0.0.1:44267/ using document root <none> and password file <none>
I20260929 21:04:39.247140 21189 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:39.247282 21189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:39.247594 21189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:39.250615 21189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "a90f697644894b6a8c365e37a6b7370b"
format_stamp: "Formatted at 2026-09-29 21:04:39 on dist-test-slave-1dms"
server_key: "e5758c2a856d6f01390b697988ce4750"
server_key_iv: "0c6a507bd6409cc7f3acce01e9d5af98"
server_key_version: "encryptionkey@0"
I20260929 21:04:39.251355 21189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "a90f697644894b6a8c365e37a6b7370b"
format_stamp: "Formatted at 2026-09-29 21:04:39 on dist-test-slave-1dms"
server_key: "e5758c2a856d6f01390b697988ce4750"
server_key_iv: "0c6a507bd6409cc7f3acce01e9d5af98"
server_key_version: "encryptionkey@0"
I20260929 21:04:39.256248 21189 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 21:04:39.259594 21204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:39.260932 21189 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:04:39.261101 21189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "a90f697644894b6a8c365e37a6b7370b"
format_stamp: "Formatted at 2026-09-29 21:04:39 on dist-test-slave-1dms"
server_key: "e5758c2a856d6f01390b697988ce4750"
server_key_iv: "0c6a507bd6409cc7f3acce01e9d5af98"
server_key_version: "encryptionkey@0"
I20260929 21:04:39.261283 21189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:39.283282 21189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:39.287631 21189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:39.287931 21189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:39.306772 21189 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40053
I20260929 21:04:39.306804 21255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40053 every 8 connection(s)
I20260929 21:04:39.308138 21189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:04:39.309531 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 21189
I20260929 21:04:39.309715 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:04:39.310266 388 external_mini_cluster.cc:1428] Setting key cf5fa600af47452b13214353a2e46d7a
I20260929 21:04:39.312909 21256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:04:39.325150 21259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:39.312041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46899 (local address 127.0.0.1:40053)
0929 21:04:39.312791 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:39.312806 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:39.312877 (+ 71us) server_negotiation.cc:407] Connection header received
0929 21:04:39.315200 (+ 2323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:39.315229 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:39.315722 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:39.316171 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:39.316673 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:39.318695 (+ 2022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:39.320059 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:39.321089 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:39.321265 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:39.322388 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:39.322398 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:39.322417 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:39.323141 (+ 724us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:39.323240 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:39.323276 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:39.323390 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:39.323616 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 21:04:39.323882 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":302,"threads_started":1}
I20260929 21:04:39.326377 21256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a90f697644894b6a8c365e37a6b7370b: Bootstrap starting.
I20260929 21:04:39.328730 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 21:04:39.329711 21256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a90f697644894b6a8c365e37a6b7370b: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:39.330884 21256 log.cc:824] T 00000000000000000000000000000000 P a90f697644894b6a8c365e37a6b7370b: Log is configured to *not* fsync() on all Append() calls
I20260929 21:04:39.334177 21256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a90f697644894b6a8c365e37a6b7370b: No bootstrap required, opened a new log
I20260929 21:04:39.340636 21259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:39.333965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48635 (local address 127.0.0.1:40053)
0929 21:04:39.334127 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:39.334134 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:39.334641 (+ 507us) server_negotiation.cc:407] Connection header received
0929 21:04:39.334826 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:39.334830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:39.334890 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:39.334968 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:39.335567 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:39.336325 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:39.338229 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:39.338669 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:39.338763 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:39.339909 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:39.339913 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:39.339916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:39.339933 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:39.339974 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:39.339988 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:39.340061 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:39.340204 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 21:04:39.340278 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 21:04:39.343855 21256 raft_consensus.cc:348] T 00000000000000000000000000000000 P a90f697644894b6a8c365e37a6b7370b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a90f697644894b6a8c365e37a6b7370b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40053 } }
I20260929 21:04:39.344215 21256 raft_consensus.cc:374] T 00000000000000000000000000000000 P a90f697644894b6a8c365e37a6b7370b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:39.344342 21256 raft_consensus.cc:729] T 00000000000000000000000000000000 P a90f697644894b6a8c365e37a6b7370b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a90f697644894b6a8c365e37a6b7370b, State: Initialized, Role: FOLLOWER
I20260929 21:04:39.344949 21256 consensus_queue.cc:260] T 00000000000000000000000000000000 P a90f697644894b6a8c365e37a6b7370b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a90f697644894b6a8c365e37a6b7370b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40053 } }
I20260929 21:04:39.345201 21256 raft_consensus.cc:388] T 00000000000000000000000000000000 P a90f697644894b6a8c365e37a6b7370b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:04:39.345350 21256 raft_consensus.cc:482] T 00000000000000000000000000000000 P a90f697644894b6a8c365e37a6b7370b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:04:39.345548 21256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a90f697644894b6a8c365e37a6b7370b [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:39.347692 21256 raft_consensus.cc:504] T 00000000000000000000000000000000 P a90f697644894b6a8c365e37a6b7370b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a90f697644894b6a8c365e37a6b7370b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40053 } }
I20260929 21:04:39.348168 21256 leader_election.cc:302] T 00000000000000000000000000000000 P a90f697644894b6a8c365e37a6b7370b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a90f697644894b6a8c365e37a6b7370b; no voters:
I20260929 21:04:39.348748 21256 leader_election.cc:288] T 00000000000000000000000000000000 P a90f697644894b6a8c365e37a6b7370b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 21:04:39.348896 21266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a90f697644894b6a8c365e37a6b7370b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:04:39.349587 21266 raft_consensus.cc:686] T 00000000000000000000000000000000 P a90f697644894b6a8c365e37a6b7370b [term 1 LEADER]: Becoming Leader. State: Replica: a90f697644894b6a8c365e37a6b7370b, State: Running, Role: LEADER
I20260929 21:04:39.350116 21266 consensus_queue.cc:237] T 00000000000000000000000000000000 P a90f697644894b6a8c365e37a6b7370b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a90f697644894b6a8c365e37a6b7370b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40053 } }
I20260929 21:04:39.350773 21256 sys_catalog.cc:564] T 00000000000000000000000000000000 P a90f697644894b6a8c365e37a6b7370b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:04:39.353291 21268 sys_catalog.cc:455] T 00000000000000000000000000000000 P a90f697644894b6a8c365e37a6b7370b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a90f697644894b6a8c365e37a6b7370b. Latest consensus state: current_term: 1 leader_uuid: "a90f697644894b6a8c365e37a6b7370b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a90f697644894b6a8c365e37a6b7370b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40053 } } }
I20260929 21:04:39.353762 21268 sys_catalog.cc:458] T 00000000000000000000000000000000 P a90f697644894b6a8c365e37a6b7370b [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:39.354552 21267 sys_catalog.cc:455] T 00000000000000000000000000000000 P a90f697644894b6a8c365e37a6b7370b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a90f697644894b6a8c365e37a6b7370b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a90f697644894b6a8c365e37a6b7370b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40053 } } }
I20260929 21:04:39.354887 21267 sys_catalog.cc:458] T 00000000000000000000000000000000 P a90f697644894b6a8c365e37a6b7370b [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:39.357882 21274 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:04:39.362661 21274 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:04:39.371935 21274 catalog_manager.cc:1347] Generated new cluster ID: 44fde0db92c54d0db80c400a4498d731
I20260929 21:04:39.372049 21274 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:04:39.408864 21274 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:04:39.410712 21274 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:04:39.429323 21274 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a90f697644894b6a8c365e37a6b7370b: Generated new TSK 0
I20260929 21:04:39.430649 21274 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 21:04:39.436573 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 21189
W20260929 21:04:39.437757 21251 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46899
W20260929 21:04:39.438478 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 21189: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 21:04:39.438634 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 21189
2026-09-29T21:04:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1898 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 21:04:39 dist-test-slave-1dms krb5kdc[21287](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 21:04:39 dist-test-slave-1dms krb5kdc[21287](info): set up 2 sockets
Sep 29 21:04:39 dist-test-slave-1dms krb5kdc[21287](info): commencing operation
krb5kdc: starting...
W20260929 21:04:40.545137 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 21:04:40 dist-test-slave-1dms krb5kdc[21287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T21:04:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:04:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 21:04:40.795603 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40671
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:04:41.113725 21300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:41.114049 21300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:41.114228 21300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:41.121249 21300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:04:41.121388 21300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:41.121490 21300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:41.121594 21300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:04:41.121688 21300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:04:41.131240 21300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:41.132474 21300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:41.133771 21300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:41.145390 21305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:41.145463 21308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:41.145426 21306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:41.146152 21300 server_base.cc:1034] running on GCE node
I20260929 21:04:41.147046 21300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:41.148526 21300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:41.149758 21300 hybrid_clock.cc:648] HybridClock initialized: now 1790715881149710 us; error 61 us; skew 500 ppm
I20260929 21:04:41.150113 21300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:04:41 dist-test-slave-1dms krb5kdc[21287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:04:41.157713 21300 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 21:04:41.159170 21300 webserver.cc:460] Webserver started at http://127.0.0.1:34687/ using document root <none> and password file <none>
I20260929 21:04:41.159832 21300 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:41.159935 21300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:41.160185 21300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:41.163213 21300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "857dea410106412d858808d973cc7d40"
format_stamp: "Formatted at 2026-09-29 21:04:41 on dist-test-slave-1dms"
server_key: "fcd92d816e9ec99a1a8da0491e6a0664"
server_key_iv: "d2605789b509425613ac49379e850ecd"
server_key_version: "encryptionkey@0"
I20260929 21:04:41.163919 21300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "857dea410106412d858808d973cc7d40"
format_stamp: "Formatted at 2026-09-29 21:04:41 on dist-test-slave-1dms"
server_key: "fcd92d816e9ec99a1a8da0491e6a0664"
server_key_iv: "d2605789b509425613ac49379e850ecd"
server_key_version: "encryptionkey@0"
I20260929 21:04:41.168991 21300 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 21:04:41.172204 21315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:41.173615 21300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:04:41.173758 21300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "857dea410106412d858808d973cc7d40"
format_stamp: "Formatted at 2026-09-29 21:04:41 on dist-test-slave-1dms"
server_key: "fcd92d816e9ec99a1a8da0491e6a0664"
server_key_iv: "d2605789b509425613ac49379e850ecd"
server_key_version: "encryptionkey@0"
I20260929 21:04:41.173944 21300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:41.179663 21300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:41.183708 21300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:41.183969 21300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:41.201990 21300 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40671
I20260929 21:04:41.202025 21366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40671 every 8 connection(s)
I20260929 21:04:41.203369 21300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:04:41.207010 21367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:04:41.208019 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 21300
I20260929 21:04:41.208209 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:04:41.208765 388 external_mini_cluster.cc:1428] Setting key d6f307ab44b4e3b030a78a6334402c4e
I20260929 21:04:41.216621 21370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:41.210584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39541 (local address 127.0.0.1:40671)
0929 21:04:41.211368 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:41.211379 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:41.211422 (+ 43us) server_negotiation.cc:407] Connection header received
0929 21:04:41.213330 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:41.213353 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:41.213687 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:41.214098 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:41.215103 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:41.215109 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:41.215122 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:41.215136 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:41.215193 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:41.215212 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:41.215446 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:41.215573 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 21:04:41.215751 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":352,"threads_started":1}
I20260929 21:04:41.219573 21367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 857dea410106412d858808d973cc7d40: Bootstrap starting.
I20260929 21:04:41.220072 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 21:04:41.222615 21367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 857dea410106412d858808d973cc7d40: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:41.223613 21367 log.cc:824] T 00000000000000000000000000000000 P 857dea410106412d858808d973cc7d40: Log is configured to *not* fsync() on all Append() calls
I20260929 21:04:41.226610 21367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 857dea410106412d858808d973cc7d40: No bootstrap required, opened a new log
I20260929 21:04:41.227123 21370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:41.224333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57243 (local address 127.0.0.1:40671)
0929 21:04:41.224527 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:41.224535 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:41.225059 (+ 524us) server_negotiation.cc:407] Connection header received
0929 21:04:41.225351 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:41.225356 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:41.225417 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:41.225518 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:41.226442 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:41.226447 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:41.226452 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:41.226469 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:41.226516 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:41.226531 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:41.226669 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:41.226797 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 21:04:41.226849 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 21:04:41.234212 21367 raft_consensus.cc:348] T 00000000000000000000000000000000 P 857dea410106412d858808d973cc7d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857dea410106412d858808d973cc7d40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40671 } }
I20260929 21:04:41.234573 21367 raft_consensus.cc:374] T 00000000000000000000000000000000 P 857dea410106412d858808d973cc7d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:41.234674 21367 raft_consensus.cc:729] T 00000000000000000000000000000000 P 857dea410106412d858808d973cc7d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 857dea410106412d858808d973cc7d40, State: Initialized, Role: FOLLOWER
I20260929 21:04:41.235188 21367 consensus_queue.cc:260] T 00000000000000000000000000000000 P 857dea410106412d858808d973cc7d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857dea410106412d858808d973cc7d40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40671 } }
I20260929 21:04:41.235419 21367 raft_consensus.cc:388] T 00000000000000000000000000000000 P 857dea410106412d858808d973cc7d40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:04:41.235558 21367 raft_consensus.cc:482] T 00000000000000000000000000000000 P 857dea410106412d858808d973cc7d40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:04:41.235718 21367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 857dea410106412d858808d973cc7d40 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:41.237906 21367 raft_consensus.cc:504] T 00000000000000000000000000000000 P 857dea410106412d858808d973cc7d40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857dea410106412d858808d973cc7d40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40671 } }
I20260929 21:04:41.238377 21367 leader_election.cc:302] T 00000000000000000000000000000000 P 857dea410106412d858808d973cc7d40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 857dea410106412d858808d973cc7d40; no voters:
I20260929 21:04:41.238984 21367 leader_election.cc:288] T 00000000000000000000000000000000 P 857dea410106412d858808d973cc7d40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 21:04:41.239101 21377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 857dea410106412d858808d973cc7d40 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:04:41.240978 21367 sys_catalog.cc:564] T 00000000000000000000000000000000 P 857dea410106412d858808d973cc7d40 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:04:41.241333 21377 raft_consensus.cc:686] T 00000000000000000000000000000000 P 857dea410106412d858808d973cc7d40 [term 1 LEADER]: Becoming Leader. State: Replica: 857dea410106412d858808d973cc7d40, State: Running, Role: LEADER
I20260929 21:04:41.241837 21377 consensus_queue.cc:237] T 00000000000000000000000000000000 P 857dea410106412d858808d973cc7d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857dea410106412d858808d973cc7d40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40671 } }
I20260929 21:04:41.244649 21378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 857dea410106412d858808d973cc7d40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "857dea410106412d858808d973cc7d40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857dea410106412d858808d973cc7d40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40671 } } }
I20260929 21:04:41.245150 21378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 857dea410106412d858808d973cc7d40 [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:41.245539 21379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 857dea410106412d858808d973cc7d40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 857dea410106412d858808d973cc7d40. Latest consensus state: current_term: 1 leader_uuid: "857dea410106412d858808d973cc7d40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857dea410106412d858808d973cc7d40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40671 } } }
I20260929 21:04:41.245788 21379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 857dea410106412d858808d973cc7d40 [sys.catalog]: This master's current role is: LEADER
W20260929 21:04:41.251798 21393 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 857dea410106412d858808d973cc7d40: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 21:04:41.251911 21393 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 21:04:41.252045 21389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:04:41.253696 21389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:04:41.261366 21389 catalog_manager.cc:1347] Generated new cluster ID: 7ff5ea99f82a456c8ae2253f9f269593
I20260929 21:04:41.261482 21389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:04:41.271364 21389 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:04:41.271760 21389 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:04:41.283653 21389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 857dea410106412d858808d973cc7d40: Generated new TSK 0
I20260929 21:04:41.284498 21389 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 21:04:41.295101 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 21300
W20260929 21:04:41.296083 21362 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39541
W20260929 21:04:41.296782 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 21300: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 21:04:41.296981 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 21300
2026-09-29T21:04:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1857 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 21:04:41 dist-test-slave-1dms krb5kdc[21398](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 21:04:41 dist-test-slave-1dms krb5kdc[21398](info): set up 2 sockets
Sep 29 21:04:41 dist-test-slave-1dms krb5kdc[21398](info): commencing operation
krb5kdc: starting...
W20260929 21:04:42.385207 388 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.011s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 21:04:42 dist-test-slave-1dms krb5kdc[21398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T21:04:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:04:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 21:04:42.636698 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34539
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:04:42.957129 21411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:42.957484 21411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:42.957685 21411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:42.965245 21411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:04:42.965389 21411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:42.965473 21411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:42.965569 21411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:04:42.965636 21411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:04:42.975363 21411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:42.976679 21411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:42.978031 21411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:42.989756 21416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:42.989897 21417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:42.989892 21419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:42.990855 21411 server_base.cc:1034] running on GCE node
I20260929 21:04:42.991643 21411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:42.993027 21411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:42.994282 21411 hybrid_clock.cc:648] HybridClock initialized: now 1790715882994246 us; error 41 us; skew 500 ppm
I20260929 21:04:42.994649 21411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:04:42 dist-test-slave-1dms krb5kdc[21398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:04:42.997789 21411 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 21:04:42.999341 21411 webserver.cc:460] Webserver started at http://127.0.0.1:40795/ using document root <none> and password file <none>
I20260929 21:04:43.000036 21411 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:43.000190 21411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:43.000504 21411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:43.003463 21411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "31245c52a83a4787951e109c34a349e3"
format_stamp: "Formatted at 2026-09-29 21:04:42 on dist-test-slave-1dms"
server_key: "642223e1c0ee4c1fca23eec3b8d9c6fe"
server_key_iv: "c71f3e2ef5352e74933e3e984c67dc73"
server_key_version: "encryptionkey@0"
I20260929 21:04:43.004246 21411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "31245c52a83a4787951e109c34a349e3"
format_stamp: "Formatted at 2026-09-29 21:04:42 on dist-test-slave-1dms"
server_key: "642223e1c0ee4c1fca23eec3b8d9c6fe"
server_key_iv: "c71f3e2ef5352e74933e3e984c67dc73"
server_key_version: "encryptionkey@0"
I20260929 21:04:43.009456 21411 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 21:04:43.013101 21426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:43.014358 21411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 21:04:43.014540 21411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "31245c52a83a4787951e109c34a349e3"
format_stamp: "Formatted at 2026-09-29 21:04:42 on dist-test-slave-1dms"
server_key: "642223e1c0ee4c1fca23eec3b8d9c6fe"
server_key_iv: "c71f3e2ef5352e74933e3e984c67dc73"
server_key_version: "encryptionkey@0"
I20260929 21:04:43.014762 21411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:43.033977 21411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:43.038174 21411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:43.038446 21411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:43.056286 21411 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34539
I20260929 21:04:43.056316 21477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34539 every 8 connection(s)
I20260929 21:04:43.057623 21411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:04:43.059944 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 21411
I20260929 21:04:43.060132 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:04:43.060660 388 external_mini_cluster.cc:1428] Setting key 4e0809cbeac46635e009c4e992f3ecd4
I20260929 21:04:43.061726 21478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 21:04:43 dist-test-slave-1dms krb5kdc[21398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 21:04:43.075246 21478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31245c52a83a4787951e109c34a349e3: Bootstrap starting.
I20260929 21:04:43.079315 21478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31245c52a83a4787951e109c34a349e3: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:43.080524 21478 log.cc:824] T 00000000000000000000000000000000 P 31245c52a83a4787951e109c34a349e3: Log is configured to *not* fsync() on all Append() calls
I20260929 21:04:43.083913 21478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31245c52a83a4787951e109c34a349e3: No bootstrap required, opened a new log
I20260929 21:04:43.084904 21481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:43.062377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:51787 (local address 127.0.0.1:34539)
0929 21:04:43.063106 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:43.063121 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:43.063179 (+ 58us) server_negotiation.cc:407] Connection header received
0929 21:04:43.065116 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:43.065140 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:43.065648 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:43.066053 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:43.067282 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:43.069342 (+ 2060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:43.070547 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:43.071531 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:43.071713 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:43.075960 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:43.075970 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:43.075989 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:04:43.076043 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:43.079988 (+ 3945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:43.080808 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:43.080814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:43.080820 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:43.081073 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:43.081726 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:43.081733 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:43.081737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:43.082463 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:43.082757 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:43.083197 (+ 440us) server_negotiation.cc:299] Negotiation successful
0929 21:04:43.083526 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":311,"threads_started":1}
I20260929 21:04:43.088627 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 21:04:43.093902 21478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31245c52a83a4787951e109c34a349e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31245c52a83a4787951e109c34a349e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34539 } }
I20260929 21:04:43.094483 21478 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31245c52a83a4787951e109c34a349e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:43.094609 21478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31245c52a83a4787951e109c34a349e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31245c52a83a4787951e109c34a349e3, State: Initialized, Role: FOLLOWER
I20260929 21:04:43.095257 21478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31245c52a83a4787951e109c34a349e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31245c52a83a4787951e109c34a349e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34539 } }
I20260929 21:04:43.095487 21478 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31245c52a83a4787951e109c34a349e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:04:43.095642 21478 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31245c52a83a4787951e109c34a349e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:04:43.095810 21478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31245c52a83a4787951e109c34a349e3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:43.098270 21478 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31245c52a83a4787951e109c34a349e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31245c52a83a4787951e109c34a349e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34539 } }
I20260929 21:04:43.098764 21478 leader_election.cc:302] T 00000000000000000000000000000000 P 31245c52a83a4787951e109c34a349e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31245c52a83a4787951e109c34a349e3; no voters:
I20260929 21:04:43.099417 21478 leader_election.cc:288] T 00000000000000000000000000000000 P 31245c52a83a4787951e109c34a349e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 21:04:43.099519 21488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31245c52a83a4787951e109c34a349e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:04:43.101033 21488 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31245c52a83a4787951e109c34a349e3 [term 1 LEADER]: Becoming Leader. State: Replica: 31245c52a83a4787951e109c34a349e3, State: Running, Role: LEADER
I20260929 21:04:43.101606 21488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31245c52a83a4787951e109c34a349e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31245c52a83a4787951e109c34a349e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34539 } }
I20260929 21:04:43.101854 21478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31245c52a83a4787951e109c34a349e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:04:43.108953 21481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:43.093927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:46289 (local address 127.0.0.1:34539)
0929 21:04:43.094165 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:43.094174 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:43.094633 (+ 459us) server_negotiation.cc:407] Connection header received
0929 21:04:43.094832 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:43.094836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:43.094900 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:43.095014 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:43.096724 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:43.097596 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:43.098894 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:43.099480 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:43.099662 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:43.102733 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:43.102741 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:43.102744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:04:43.102773 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:43.106087 (+ 3314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:43.106794 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:43.106800 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:43.106804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:43.106930 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:43.107366 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:43.107371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:43.107374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:43.107719 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:43.108123 (+ 404us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:43.108317 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 21:04:43.108493 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 21:04:43.110436 21489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31245c52a83a4787951e109c34a349e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31245c52a83a4787951e109c34a349e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31245c52a83a4787951e109c34a349e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34539 } } }
I20260929 21:04:43.110886 21489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31245c52a83a4787951e109c34a349e3 [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:43.111904 21490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31245c52a83a4787951e109c34a349e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31245c52a83a4787951e109c34a349e3. Latest consensus state: current_term: 1 leader_uuid: "31245c52a83a4787951e109c34a349e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31245c52a83a4787951e109c34a349e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34539 } } }
I20260929 21:04:43.113183 21490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31245c52a83a4787951e109c34a349e3 [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:43.113854 21496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:04:43.118719 21496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:04:43.131455 21496 catalog_manager.cc:1347] Generated new cluster ID: bf22d309808d45b7ab51be68ce736c3a
I20260929 21:04:43.131577 21496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:04:43.141083 21496 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:04:43.142365 21496 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:04:43.157615 21496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31245c52a83a4787951e109c34a349e3: Generated new TSK 0
I20260929 21:04:43.158473 21496 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 21:04:43.164212 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 21411
I20260929 21:04:43.165303 21473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:04:43.256415 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 21411
W20260929 21:04:43.275799 21396 connection.cc:535] client connection to 127.0.0.1:34539 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34539): Connection reset by peer (error 104)
2026-09-29T21:04:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1962 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 21:04:43 dist-test-slave-1dms krb5kdc[21511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 21:04:43 dist-test-slave-1dms krb5kdc[21511](info): set up 2 sockets
Sep 29 21:04:43 dist-test-slave-1dms krb5kdc[21511](info): commencing operation
krb5kdc: starting...
W20260929 21:04:44.365267 388 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.005s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 21:04:44 dist-test-slave-1dms krb5kdc[21511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T21:04:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:04:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 21:04:44.632354 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38777
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:04:44.965094 21524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:44.965387 21524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:44.965480 21524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:44.972602 21524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:04:44.972712 21524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:44.972766 21524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:44.972826 21524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:04:44.972936 21524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:04:44.982826 21524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:44.984047 21524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:44.985443 21524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:44.996956 21529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:44.997030 21532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:44.996922 21530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:44.997843 21524 server_base.cc:1034] running on GCE node
I20260929 21:04:44.998581 21524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:44.999922 21524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:45.001119 21524 hybrid_clock.cc:648] HybridClock initialized: now 1790715885001061 us; error 54 us; skew 500 ppm
I20260929 21:04:45.001471 21524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:04:44 dist-test-slave-1dms krb5kdc[21511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:04:45.004645 21524 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 21:04:45.006142 21524 webserver.cc:460] Webserver started at http://127.0.0.1:39663/ using document root <none> and password file <none>
I20260929 21:04:45.006808 21524 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:45.006918 21524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:45.007179 21524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:45.010131 21524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "db9b1726a41641cbb24f00ce0278b436"
format_stamp: "Formatted at 2026-09-29 21:04:45 on dist-test-slave-1dms"
server_key: "c80a7d4bb40d16e096ce809524d78fb8"
server_key_iv: "cb5972227e32a34d04580252e57b8c95"
server_key_version: "encryptionkey@0"
I20260929 21:04:45.010826 21524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "db9b1726a41641cbb24f00ce0278b436"
format_stamp: "Formatted at 2026-09-29 21:04:45 on dist-test-slave-1dms"
server_key: "c80a7d4bb40d16e096ce809524d78fb8"
server_key_iv: "cb5972227e32a34d04580252e57b8c95"
server_key_version: "encryptionkey@0"
I20260929 21:04:45.015911 21524 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260929 21:04:45.019341 21539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:45.020771 21524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 21:04:45.020982 21524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "db9b1726a41641cbb24f00ce0278b436"
format_stamp: "Formatted at 2026-09-29 21:04:45 on dist-test-slave-1dms"
server_key: "c80a7d4bb40d16e096ce809524d78fb8"
server_key_iv: "cb5972227e32a34d04580252e57b8c95"
server_key_version: "encryptionkey@0"
I20260929 21:04:45.021196 21524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:45.043138 21524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:45.048195 21524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:45.048501 21524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:45.067289 21524 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38777
I20260929 21:04:45.067345 21590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38777 every 8 connection(s)
I20260929 21:04:45.068732 21524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:04:45.072641 21591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:04:45.076341 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 21524
I20260929 21:04:45.076547 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:04:45.077034 388 external_mini_cluster.cc:1428] Setting key e22057619e273ccabce4aabf0efda592
I20260929 21:04:45.084652 21591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db9b1726a41641cbb24f00ce0278b436: Bootstrap starting.
I20260929 21:04:45.087807 21591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db9b1726a41641cbb24f00ce0278b436: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:45.088932 21591 log.cc:824] T 00000000000000000000000000000000 P db9b1726a41641cbb24f00ce0278b436: Log is configured to *not* fsync() on all Append() calls
I20260929 21:04:45.090057 21594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:45.078782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:36369 (local address 127.0.0.1:38777)
0929 21:04:45.079498 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:45.079509 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:45.079549 (+ 40us) server_negotiation.cc:407] Connection header received
0929 21:04:45.081294 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:45.081318 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:45.081766 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:45.082157 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:45.082736 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:45.084010 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:45.085482 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:45.086183 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:45.086381 (+ 198us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:45.087654 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:45.087662 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:45.087676 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:45.088287 (+ 611us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:45.088359 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:45.088383 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:45.088545 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:45.088852 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 21:04:45.089015 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":318,"threads_started":1}
I20260929 21:04:45.091763 21591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db9b1726a41641cbb24f00ce0278b436: No bootstrap required, opened a new log
I20260929 21:04:45.093818 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 21:04:45.100673 21591 raft_consensus.cc:348] T 00000000000000000000000000000000 P db9b1726a41641cbb24f00ce0278b436 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9b1726a41641cbb24f00ce0278b436" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38777 } }
I20260929 21:04:45.101135 21591 raft_consensus.cc:374] T 00000000000000000000000000000000 P db9b1726a41641cbb24f00ce0278b436 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:45.101253 21591 raft_consensus.cc:729] T 00000000000000000000000000000000 P db9b1726a41641cbb24f00ce0278b436 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db9b1726a41641cbb24f00ce0278b436, State: Initialized, Role: FOLLOWER
I20260929 21:04:45.101971 21591 consensus_queue.cc:260] T 00000000000000000000000000000000 P db9b1726a41641cbb24f00ce0278b436 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9b1726a41641cbb24f00ce0278b436" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38777 } }
I20260929 21:04:45.102258 21591 raft_consensus.cc:388] T 00000000000000000000000000000000 P db9b1726a41641cbb24f00ce0278b436 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:04:45.102407 21591 raft_consensus.cc:482] T 00000000000000000000000000000000 P db9b1726a41641cbb24f00ce0278b436 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:04:45.102567 21591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db9b1726a41641cbb24f00ce0278b436 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:45.105669 21591 raft_consensus.cc:504] T 00000000000000000000000000000000 P db9b1726a41641cbb24f00ce0278b436 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9b1726a41641cbb24f00ce0278b436" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38777 } }
I20260929 21:04:45.105980 21594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:45.098216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:54485 (local address 127.0.0.1:38777)
0929 21:04:45.098405 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:45.098417 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:45.099071 (+ 654us) server_negotiation.cc:407] Connection header received
0929 21:04:45.099431 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:45.099435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:45.099508 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:45.099681 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:45.100290 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:45.101245 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:45.103201 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:45.103733 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:45.103835 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:45.105114 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:45.105120 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:45.105124 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:45.105144 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:45.105203 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:45.105219 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:45.105311 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:45.105541 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 21:04:45.105585 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 21:04:45.106308 21591 leader_election.cc:302] T 00000000000000000000000000000000 P db9b1726a41641cbb24f00ce0278b436 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db9b1726a41641cbb24f00ce0278b436; no voters:
I20260929 21:04:45.107007 21591 leader_election.cc:288] T 00000000000000000000000000000000 P db9b1726a41641cbb24f00ce0278b436 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 21:04:45.107121 21601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db9b1726a41641cbb24f00ce0278b436 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:04:45.108981 21601 raft_consensus.cc:686] T 00000000000000000000000000000000 P db9b1726a41641cbb24f00ce0278b436 [term 1 LEADER]: Becoming Leader. State: Replica: db9b1726a41641cbb24f00ce0278b436, State: Running, Role: LEADER
I20260929 21:04:45.109534 21601 consensus_queue.cc:237] T 00000000000000000000000000000000 P db9b1726a41641cbb24f00ce0278b436 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9b1726a41641cbb24f00ce0278b436" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38777 } }
I20260929 21:04:45.109754 21591 sys_catalog.cc:564] T 00000000000000000000000000000000 P db9b1726a41641cbb24f00ce0278b436 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:04:45.112352 21603 sys_catalog.cc:455] T 00000000000000000000000000000000 P db9b1726a41641cbb24f00ce0278b436 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db9b1726a41641cbb24f00ce0278b436. Latest consensus state: current_term: 1 leader_uuid: "db9b1726a41641cbb24f00ce0278b436" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9b1726a41641cbb24f00ce0278b436" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38777 } } }
I20260929 21:04:45.112802 21603 sys_catalog.cc:458] T 00000000000000000000000000000000 P db9b1726a41641cbb24f00ce0278b436 [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:45.112365 21602 sys_catalog.cc:455] T 00000000000000000000000000000000 P db9b1726a41641cbb24f00ce0278b436 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db9b1726a41641cbb24f00ce0278b436" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9b1726a41641cbb24f00ce0278b436" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38777 } } }
I20260929 21:04:45.113235 21602 sys_catalog.cc:458] T 00000000000000000000000000000000 P db9b1726a41641cbb24f00ce0278b436 [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:45.117691 21607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:04:45.122864 21607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:04:45.129688 21618 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 21:04:45.131635 21607 catalog_manager.cc:1347] Generated new cluster ID: 66c436001d2c474ebbb20fb3d6c53d4d
I20260929 21:04:45.131738 21607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:04:45.150182 21607 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:04:45.152158 21607 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:04:45.163039 21607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db9b1726a41641cbb24f00ce0278b436: Generated new TSK 0
I20260929 21:04:45.164162 21607 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 21:04:45.177749 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 21524
W20260929 21:04:45.178978 21586 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.108:36369
W20260929 21:04:45.179737 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 21524: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 21:04:45.179885 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 21524
W20260929 21:04:45.197180 21509 connection.cc:535] client connection to 127.0.0.1:38777 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38777): Connection reset by peer (error 104)
2026-09-29T21:04:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1921 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 21:04:45 dist-test-slave-1dms krb5kdc[21622](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 21:04:45 dist-test-slave-1dms krb5kdc[21622](info): set up 2 sockets
Sep 29 21:04:45 dist-test-slave-1dms krb5kdc[21622](info): commencing operation
krb5kdc: starting...
W20260929 21:04:46.289304 388 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 21:04:46 dist-test-slave-1dms krb5kdc[21622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T21:04:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:04:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 21:04:46.549952 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34651
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:04:46.878971 21635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:46.879263 21635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:46.879424 21635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:46.886823 21635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:04:46.886940 21635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:46.887053 21635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:46.887162 21635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:04:46.887243 21635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:04:46.897433 21635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:46.898792 21635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:46.900132 21635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:46.912225 21643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:46.912370 21641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:46.912532 21640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:46.913595 21635 server_base.cc:1034] running on GCE node
I20260929 21:04:46.914477 21635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:46.915951 21635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:46.917194 21635 hybrid_clock.cc:648] HybridClock initialized: now 1790715886917128 us; error 37 us; skew 500 ppm
I20260929 21:04:46.917626 21635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:04:46 dist-test-slave-1dms krb5kdc[21622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:04:46.926553 21635 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 21:04:46.928469 21635 webserver.cc:460] Webserver started at http://127.0.0.1:34207/ using document root <none> and password file <none>
I20260929 21:04:46.929286 21635 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:46.929437 21635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:46.929766 21635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:46.932775 21635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "66284dfac9e74fc4b134d20b097caa20"
format_stamp: "Formatted at 2026-09-29 21:04:46 on dist-test-slave-1dms"
server_key: "945be731b39978bb178ba5f419f47581"
server_key_iv: "a03b01b130c449a672df50c313e9f812"
server_key_version: "encryptionkey@0"
I20260929 21:04:46.933699 21635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "66284dfac9e74fc4b134d20b097caa20"
format_stamp: "Formatted at 2026-09-29 21:04:46 on dist-test-slave-1dms"
server_key: "945be731b39978bb178ba5f419f47581"
server_key_iv: "a03b01b130c449a672df50c313e9f812"
server_key_version: "encryptionkey@0"
I20260929 21:04:46.939002 21635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 21:04:46.942754 21650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:46.944300 21635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 21:04:46.944492 21635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "66284dfac9e74fc4b134d20b097caa20"
format_stamp: "Formatted at 2026-09-29 21:04:46 on dist-test-slave-1dms"
server_key: "945be731b39978bb178ba5f419f47581"
server_key_iv: "a03b01b130c449a672df50c313e9f812"
server_key_version: "encryptionkey@0"
I20260929 21:04:46.944723 21635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:46.950649 21635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:46.955099 21635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:46.955410 21635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:46.974004 21635 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34651
I20260929 21:04:46.974042 21701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34651 every 8 connection(s)
I20260929 21:04:46.975445 21635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:04:46.979303 21702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:04:46.984740 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 21635
I20260929 21:04:46.984992 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:04:46.985437 388 external_mini_cluster.cc:1428] Setting key be71cd1b99b352913da18fde33de5fab
I20260929 21:04:46.990088 21702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66284dfac9e74fc4b134d20b097caa20: Bootstrap starting.
I20260929 21:04:46.993573 21705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:46.987322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:44413 (local address 127.0.0.1:34651)
0929 21:04:46.988181 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:46.988193 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:46.988236 (+ 43us) server_negotiation.cc:407] Connection header received
0929 21:04:46.989995 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:46.990026 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:46.990436 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:46.990772 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:46.991896 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:46.991902 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:46.991917 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:46.991932 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:46.991994 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:46.992012 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:46.992176 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:46.992326 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 21:04:46.992548 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":397,"threads_started":1}
I20260929 21:04:46.994035 21702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66284dfac9e74fc4b134d20b097caa20: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:46.995276 21702 log.cc:824] T 00000000000000000000000000000000 P 66284dfac9e74fc4b134d20b097caa20: Log is configured to *not* fsync() on all Append() calls
I20260929 21:04:46.997548 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 21:04:46.998265 21702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66284dfac9e74fc4b134d20b097caa20: No bootstrap required, opened a new log
I20260929 21:04:47.004668 21705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:47.002040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:43559 (local address 127.0.0.1:34651)
0929 21:04:47.002224 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:47.002231 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:47.002727 (+ 496us) server_negotiation.cc:407] Connection header received
0929 21:04:47.002897 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:47.002901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:47.002953 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:47.003031 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:47.003990 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:47.003994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:47.003997 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:47.004012 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:47.004054 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:47.004070 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:47.004174 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:47.004299 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 21:04:47.004347 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 21:04:47.006726 21702 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66284dfac9e74fc4b134d20b097caa20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66284dfac9e74fc4b134d20b097caa20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34651 } }
I20260929 21:04:47.007115 21702 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66284dfac9e74fc4b134d20b097caa20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:47.007215 21702 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66284dfac9e74fc4b134d20b097caa20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66284dfac9e74fc4b134d20b097caa20, State: Initialized, Role: FOLLOWER
I20260929 21:04:47.007754 21702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66284dfac9e74fc4b134d20b097caa20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66284dfac9e74fc4b134d20b097caa20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34651 } }
I20260929 21:04:47.007967 21702 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66284dfac9e74fc4b134d20b097caa20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:04:47.008066 21702 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66284dfac9e74fc4b134d20b097caa20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:04:47.008203 21702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66284dfac9e74fc4b134d20b097caa20 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:47.010762 21702 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66284dfac9e74fc4b134d20b097caa20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66284dfac9e74fc4b134d20b097caa20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34651 } }
I20260929 21:04:47.011224 21702 leader_election.cc:302] T 00000000000000000000000000000000 P 66284dfac9e74fc4b134d20b097caa20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66284dfac9e74fc4b134d20b097caa20; no voters:
I20260929 21:04:47.011754 21702 leader_election.cc:288] T 00000000000000000000000000000000 P 66284dfac9e74fc4b134d20b097caa20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 21:04:47.011886 21712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66284dfac9e74fc4b134d20b097caa20 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:04:47.012603 21712 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66284dfac9e74fc4b134d20b097caa20 [term 1 LEADER]: Becoming Leader. State: Replica: 66284dfac9e74fc4b134d20b097caa20, State: Running, Role: LEADER
I20260929 21:04:47.013123 21712 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66284dfac9e74fc4b134d20b097caa20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66284dfac9e74fc4b134d20b097caa20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34651 } }
I20260929 21:04:47.013973 21702 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66284dfac9e74fc4b134d20b097caa20 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:04:47.016238 21714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66284dfac9e74fc4b134d20b097caa20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66284dfac9e74fc4b134d20b097caa20. Latest consensus state: current_term: 1 leader_uuid: "66284dfac9e74fc4b134d20b097caa20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66284dfac9e74fc4b134d20b097caa20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34651 } } }
I20260929 21:04:47.016707 21714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66284dfac9e74fc4b134d20b097caa20 [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:47.021452 21713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66284dfac9e74fc4b134d20b097caa20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66284dfac9e74fc4b134d20b097caa20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66284dfac9e74fc4b134d20b097caa20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34651 } } }
I20260929 21:04:47.021852 21718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:04:47.021835 21713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66284dfac9e74fc4b134d20b097caa20 [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:47.027168 21718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:04:47.037228 21718 catalog_manager.cc:1347] Generated new cluster ID: 4b16836262bd4e94bba79ebfcfa58ada
I20260929 21:04:47.037333 21718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:04:47.069054 21718 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:04:47.069558 21718 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:04:47.078758 21718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66284dfac9e74fc4b134d20b097caa20: Generated new TSK 0
I20260929 21:04:47.079615 21718 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 21:04:47.091264 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 21635
W20260929 21:04:47.092445 21697 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.108:44413
W20260929 21:04:47.093231 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 21635: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 21:04:47.093393 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 21635
2026-09-29T21:04:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1913 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 21:04:47 dist-test-slave-1dms krb5kdc[21733](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 21:04:47 dist-test-slave-1dms krb5kdc[21733](info): set up 2 sockets
Sep 29 21:04:47 dist-test-slave-1dms krb5kdc[21733](info): commencing operation
krb5kdc: starting...
W20260929 21:04:48.189363 388 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 21:04:48 dist-test-slave-1dms krb5kdc[21733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T21:04:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:04:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 21:04:48.448657 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46571
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:04:48.790867 21746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:48.791165 21746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:48.791257 21746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:48.798517 21746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:04:48.798645 21746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:48.798712 21746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:48.798779 21746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:04:48.798825 21746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:04:48.808632 21746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46571
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:48.810034 21746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:48.811378 21746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:48.823143 21751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:48.823184 21754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:48.823101 21752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:48.823765 21746 server_base.cc:1034] running on GCE node
I20260929 21:04:48.824585 21746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:48.826174 21746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:48.827410 21746 hybrid_clock.cc:648] HybridClock initialized: now 1790715888827369 us; error 64 us; skew 500 ppm
I20260929 21:04:48.827757 21746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:04:48 dist-test-slave-1dms krb5kdc[21733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:04:48.830865 21746 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 21:04:48.832382 21746 webserver.cc:460] Webserver started at http://127.0.0.1:39297/ using document root <none> and password file <none>
I20260929 21:04:48.833138 21746 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:48.833283 21746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:48.833585 21746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:48.836481 21746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "d0b260bc743e44c2989dfb858dc55af8"
format_stamp: "Formatted at 2026-09-29 21:04:48 on dist-test-slave-1dms"
server_key: "60fac02e23b61ae0f47cb13fb3fe8030"
server_key_iv: "bbdfa40d2b38a13deea421114791df27"
server_key_version: "encryptionkey@0"
I20260929 21:04:48.837287 21746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "d0b260bc743e44c2989dfb858dc55af8"
format_stamp: "Formatted at 2026-09-29 21:04:48 on dist-test-slave-1dms"
server_key: "60fac02e23b61ae0f47cb13fb3fe8030"
server_key_iv: "bbdfa40d2b38a13deea421114791df27"
server_key_version: "encryptionkey@0"
I20260929 21:04:48.842417 21746 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 21:04:48.845974 21761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:48.847328 21746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 21:04:48.847520 21746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "d0b260bc743e44c2989dfb858dc55af8"
format_stamp: "Formatted at 2026-09-29 21:04:48 on dist-test-slave-1dms"
server_key: "60fac02e23b61ae0f47cb13fb3fe8030"
server_key_iv: "bbdfa40d2b38a13deea421114791df27"
server_key_version: "encryptionkey@0"
I20260929 21:04:48.847719 21746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:48.864473 21746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:48.869282 21746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:48.869582 21746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:48.889356 21746 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46571
I20260929 21:04:48.889398 21812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46571 every 8 connection(s)
I20260929 21:04:48.890827 21746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:04:48.894770 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 21746
I20260929 21:04:48.894930 21813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:04:48.894961 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:04:48.895536 388 external_mini_cluster.cc:1428] Setting key 4ad0ea04099c30cade569b1599d4aa1a
I20260929 21:04:48.910000 21813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0b260bc743e44c2989dfb858dc55af8: Bootstrap starting.
Sep 29 21:04:48 dist-test-slave-1dms krb5kdc[21733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 21:04:48.913374 21813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0b260bc743e44c2989dfb858dc55af8: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:48.914495 21813 log.cc:824] T 00000000000000000000000000000000 P d0b260bc743e44c2989dfb858dc55af8: Log is configured to *not* fsync() on all Append() calls
I20260929 21:04:48.917459 21813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0b260bc743e44c2989dfb858dc55af8: No bootstrap required, opened a new log
I20260929 21:04:48.919266 21816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:48.897387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:34429 (local address 127.0.0.1:46571)
0929 21:04:48.898204 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:48.898215 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:48.898262 (+ 47us) server_negotiation.cc:407] Connection header received
0929 21:04:48.899833 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:48.899855 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:48.900278 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:48.900611 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:48.902468 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:48.903758 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:48.905603 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:48.906307 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:48.906444 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:48.911425 (+ 4981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:48.911435 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:48.911450 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:04:48.911499 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:48.914864 (+ 3365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:48.915668 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:48.915675 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:48.915681 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:48.915842 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:48.916329 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:48.916333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:48.916335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:48.916910 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:48.917252 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:48.917757 (+ 505us) server_negotiation.cc:299] Negotiation successful
0929 21:04:48.918151 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":374,"threads_started":1}
I20260929 21:04:48.923257 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 21:04:48.925972 21813 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0b260bc743e44c2989dfb858dc55af8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b260bc743e44c2989dfb858dc55af8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46571 } }
I20260929 21:04:48.926478 21813 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0b260bc743e44c2989dfb858dc55af8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:48.926620 21813 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0b260bc743e44c2989dfb858dc55af8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0b260bc743e44c2989dfb858dc55af8, State: Initialized, Role: FOLLOWER
I20260929 21:04:48.927202 21813 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0b260bc743e44c2989dfb858dc55af8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b260bc743e44c2989dfb858dc55af8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46571 } }
I20260929 21:04:48.927461 21813 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0b260bc743e44c2989dfb858dc55af8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:04:48.927635 21813 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0b260bc743e44c2989dfb858dc55af8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:04:48.927829 21813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0b260bc743e44c2989dfb858dc55af8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:48.930426 21813 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0b260bc743e44c2989dfb858dc55af8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b260bc743e44c2989dfb858dc55af8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46571 } }
I20260929 21:04:48.931159 21813 leader_election.cc:302] T 00000000000000000000000000000000 P d0b260bc743e44c2989dfb858dc55af8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0b260bc743e44c2989dfb858dc55af8; no voters:
I20260929 21:04:48.931852 21813 leader_election.cc:288] T 00000000000000000000000000000000 P d0b260bc743e44c2989dfb858dc55af8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 21:04:48.931988 21823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0b260bc743e44c2989dfb858dc55af8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:04:48.934002 21813 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0b260bc743e44c2989dfb858dc55af8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:04:48.934310 21823 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0b260bc743e44c2989dfb858dc55af8 [term 1 LEADER]: Becoming Leader. State: Replica: d0b260bc743e44c2989dfb858dc55af8, State: Running, Role: LEADER
I20260929 21:04:48.935019 21823 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0b260bc743e44c2989dfb858dc55af8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b260bc743e44c2989dfb858dc55af8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46571 } }
I20260929 21:04:48.938041 21824 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0b260bc743e44c2989dfb858dc55af8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0b260bc743e44c2989dfb858dc55af8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b260bc743e44c2989dfb858dc55af8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46571 } } }
I20260929 21:04:48.938418 21824 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0b260bc743e44c2989dfb858dc55af8 [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:48.939688 21825 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0b260bc743e44c2989dfb858dc55af8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0b260bc743e44c2989dfb858dc55af8. Latest consensus state: current_term: 1 leader_uuid: "d0b260bc743e44c2989dfb858dc55af8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b260bc743e44c2989dfb858dc55af8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46571 } } }
I20260929 21:04:48.948993 21825 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0b260bc743e44c2989dfb858dc55af8 [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:48.948827 21816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:48.927676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:34427 (local address 127.0.0.1:46571)
0929 21:04:48.927897 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:48.927905 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:48.928291 (+ 386us) server_negotiation.cc:407] Connection header received
0929 21:04:48.928462 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:48.928470 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:48.928562 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:48.928676 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:48.930541 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:48.931722 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:48.934004 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:48.934702 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:48.934831 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:48.938460 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:48.938467 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:48.938472 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:04:48.938514 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:48.943424 (+ 4910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:48.944715 (+ 1291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:48.944724 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:48.944730 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:48.944933 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:48.945658 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:48.945666 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:48.945672 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:48.946227 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:48.946609 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:48.948015 (+ 1406us) server_negotiation.cc:299] Negotiation successful
0929 21:04:48.948274 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 21:04:48.950104 21828 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:04:48.955547 21828 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:04:48.967583 21828 catalog_manager.cc:1347] Generated new cluster ID: 8fc68145e314467b88db69f4156b2212
I20260929 21:04:48.967742 21828 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:04:48.995193 21828 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:04:48.997121 21828 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:04:49.004248 21828 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0b260bc743e44c2989dfb858dc55af8: Generated new TSK 0
I20260929 21:04:49.005163 21828 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 21:04:49.017431 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 21746
I20260929 21:04:49.018487 21808 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:04:49.112591 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 21746
W20260929 21:04:49.129633 21731 connection.cc:535] client connection to 127.0.0.1:46571 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46571): Connection reset by peer (error 104)
2026-09-29T21:04:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2018 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 21:04:49 dist-test-slave-1dms krb5kdc[21845](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 21:04:49 dist-test-slave-1dms krb5kdc[21845](info): set up 2 sockets
Sep 29 21:04:49 dist-test-slave-1dms krb5kdc[21845](info): commencing operation
krb5kdc: starting...
W20260929 21:04:50.213232 388 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 21:04:50 dist-test-slave-1dms krb5kdc[21845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T21:04:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:04:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 21:04:50.456183 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43837
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:04:50.779965 21858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:50.780246 21858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:50.780409 21858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:50.787408 21858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:04:50.787513 21858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:50.787611 21858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:50.787720 21858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:04:50.787794 21858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:04:50.797194 21858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43837
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:50.798391 21858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:50.799654 21858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:50.811167 21864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:50.811126 21866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:50.811076 21863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:50.812343 21858 server_base.cc:1034] running on GCE node
I20260929 21:04:50.813066 21858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:50.814369 21858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:50.815560 21858 hybrid_clock.cc:648] HybridClock initialized: now 1790715890815523 us; error 27 us; skew 500 ppm
I20260929 21:04:50.815896 21858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:04:50 dist-test-slave-1dms krb5kdc[21845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:04:50.818902 21858 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 21:04:50.820219 21858 webserver.cc:460] Webserver started at http://127.0.0.1:46129/ using document root <none> and password file <none>
I20260929 21:04:50.820936 21858 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:50.821062 21858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:50.821368 21858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:50.824159 21858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "2a0b4131e57a4155b638bc12290f557b"
format_stamp: "Formatted at 2026-09-29 21:04:50 on dist-test-slave-1dms"
server_key: "c366c73a00660908ab04e374446976fe"
server_key_iv: "b3d2d7e09de98adce4577f3c4ec0dbad"
server_key_version: "encryptionkey@0"
I20260929 21:04:50.824934 21858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "2a0b4131e57a4155b638bc12290f557b"
format_stamp: "Formatted at 2026-09-29 21:04:50 on dist-test-slave-1dms"
server_key: "c366c73a00660908ab04e374446976fe"
server_key_iv: "b3d2d7e09de98adce4577f3c4ec0dbad"
server_key_version: "encryptionkey@0"
I20260929 21:04:50.829741 21858 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 21:04:50.832963 21873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:50.834134 21858 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 21:04:50.834290 21858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "2a0b4131e57a4155b638bc12290f557b"
format_stamp: "Formatted at 2026-09-29 21:04:50 on dist-test-slave-1dms"
server_key: "c366c73a00660908ab04e374446976fe"
server_key_iv: "b3d2d7e09de98adce4577f3c4ec0dbad"
server_key_version: "encryptionkey@0"
I20260929 21:04:50.834455 21858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:50.846398 21858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:50.850292 21858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:50.850555 21858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:50.868793 21858 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43837
I20260929 21:04:50.868886 21924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43837 every 8 connection(s)
I20260929 21:04:50.870184 21858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:04:50.873799 21925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:04:50.878763 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 21858
I20260929 21:04:50.878945 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:04:50.879449 388 external_mini_cluster.cc:1428] Setting key e94ced102a4c2322812ec95e6e435cd4
I20260929 21:04:50.884884 21925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a0b4131e57a4155b638bc12290f557b: Bootstrap starting.
I20260929 21:04:50.888391 21925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a0b4131e57a4155b638bc12290f557b: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:50.889637 21925 log.cc:824] T 00000000000000000000000000000000 P 2a0b4131e57a4155b638bc12290f557b: Log is configured to *not* fsync() on all Append() calls
I20260929 21:04:50.893011 21925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a0b4131e57a4155b638bc12290f557b: No bootstrap required, opened a new log
I20260929 21:04:50.893098 21928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:50.881242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:34717 (local address 127.0.0.1:43837)
0929 21:04:50.882024 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:50.882034 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:50.882072 (+ 38us) server_negotiation.cc:407] Connection header received
0929 21:04:50.883762 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:50.883783 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:50.884141 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:50.884439 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:50.885107 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:50.886389 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:50.888280 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:50.889060 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:50.889187 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:50.890580 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:50.890592 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:50.890614 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:50.891398 (+ 784us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:50.891467 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:50.891490 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:50.891601 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:50.891884 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 21:04:50.892081 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":378,"threads_started":1}
I20260929 21:04:50.897001 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 21:04:50.903051 21925 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a0b4131e57a4155b638bc12290f557b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0b4131e57a4155b638bc12290f557b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43837 } }
I20260929 21:04:50.903431 21925 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a0b4131e57a4155b638bc12290f557b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:50.903551 21925 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a0b4131e57a4155b638bc12290f557b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a0b4131e57a4155b638bc12290f557b, State: Initialized, Role: FOLLOWER
I20260929 21:04:50.904232 21925 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a0b4131e57a4155b638bc12290f557b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0b4131e57a4155b638bc12290f557b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43837 } }
I20260929 21:04:50.904538 21925 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a0b4131e57a4155b638bc12290f557b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:04:50.904685 21925 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a0b4131e57a4155b638bc12290f557b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:04:50.904927 21925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a0b4131e57a4155b638bc12290f557b [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:50.908003 21925 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a0b4131e57a4155b638bc12290f557b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0b4131e57a4155b638bc12290f557b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43837 } }
I20260929 21:04:50.908568 21925 leader_election.cc:302] T 00000000000000000000000000000000 P 2a0b4131e57a4155b638bc12290f557b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a0b4131e57a4155b638bc12290f557b; no voters:
I20260929 21:04:50.909711 21925 leader_election.cc:288] T 00000000000000000000000000000000 P 2a0b4131e57a4155b638bc12290f557b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 21:04:50.909881 21935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a0b4131e57a4155b638bc12290f557b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:04:50.910669 21928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:50.902136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:46541 (local address 127.0.0.1:43837)
0929 21:04:50.902321 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:50.902326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:50.903904 (+ 1578us) server_negotiation.cc:407] Connection header received
0929 21:04:50.904075 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:50.904078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:50.904141 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:50.904226 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:50.904934 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:50.905766 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:50.907646 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:50.908075 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:50.908240 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:50.909584 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:50.909592 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:50.909599 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:50.909628 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:50.909678 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:50.909703 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:50.909860 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:50.910145 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 21:04:50.910210 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":65}
I20260929 21:04:50.912081 21925 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a0b4131e57a4155b638bc12290f557b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:04:50.914463 21935 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a0b4131e57a4155b638bc12290f557b [term 1 LEADER]: Becoming Leader. State: Replica: 2a0b4131e57a4155b638bc12290f557b, State: Running, Role: LEADER
I20260929 21:04:50.915270 21935 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a0b4131e57a4155b638bc12290f557b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0b4131e57a4155b638bc12290f557b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43837 } }
I20260929 21:04:50.918377 21937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a0b4131e57a4155b638bc12290f557b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a0b4131e57a4155b638bc12290f557b. Latest consensus state: current_term: 1 leader_uuid: "2a0b4131e57a4155b638bc12290f557b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0b4131e57a4155b638bc12290f557b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43837 } } }
I20260929 21:04:50.918874 21937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a0b4131e57a4155b638bc12290f557b [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:50.918932 21936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a0b4131e57a4155b638bc12290f557b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a0b4131e57a4155b638bc12290f557b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0b4131e57a4155b638bc12290f557b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43837 } } }
I20260929 21:04:50.919234 21936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a0b4131e57a4155b638bc12290f557b [sys.catalog]: This master's current role is: LEADER
W20260929 21:04:50.924760 21949 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2a0b4131e57a4155b638bc12290f557b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 21:04:50.924913 21949 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 21:04:50.925074 21950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:04:50.926908 21950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:04:50.935653 21950 catalog_manager.cc:1347] Generated new cluster ID: 86b84534348f4f83a635c51ecb72ac51
I20260929 21:04:50.935798 21950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:04:50.964871 21950 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:04:50.966126 21950 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:04:50.979403 21950 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a0b4131e57a4155b638bc12290f557b: Generated new TSK 0
I20260929 21:04:50.980196 21950 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 21:04:50.987871 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 21858
W20260929 21:04:50.988993 21920 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.108:34717
W20260929 21:04:50.989691 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 21858: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 21:04:50.989837 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 21858
W20260929 21:04:51.006488 21843 connection.cc:535] client connection to 127.0.0.1:43837 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43837): Connection reset by peer (error 104)
2026-09-29T21:04:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1876 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 21:04:51 dist-test-slave-1dms krb5kdc[21956](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 21:04:51 dist-test-slave-1dms krb5kdc[21956](info): set up 2 sockets
Sep 29 21:04:51 dist-test-slave-1dms krb5kdc[21956](info): commencing operation
krb5kdc: starting...
W20260929 21:04:52.105296 388 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 21:04:52 dist-test-slave-1dms krb5kdc[21956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T21:04:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:04:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 21:04:52.357163 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38761
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:04:52.685868 21969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:52.686208 21969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:52.686365 21969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:52.694144 21969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:04:52.694296 21969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:52.694397 21969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:52.694501 21969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:04:52.694576 21969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:04:52.704286 21969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38761
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715556572573-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:52.705682 21969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:52.706980 21969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:52.718397 21974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:52.718425 21977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:52.718470 21975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:52.719575 21969 server_base.cc:1034] running on GCE node
I20260929 21:04:52.720335 21969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:52.721714 21969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:52.722944 21969 hybrid_clock.cc:648] HybridClock initialized: now 1790715892722907 us; error 50 us; skew 500 ppm
I20260929 21:04:52.723285 21969 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:04:52 dist-test-slave-1dms krb5kdc[21956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715892, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:04:52.730314 21969 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 21:04:52.731822 21969 webserver.cc:460] Webserver started at http://127.0.0.1:44623/ using document root <none> and password file <none>
I20260929 21:04:52.732504 21969 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:52.732651 21969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:52.733044 21969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:52.735889 21969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "0e8efd24384f42bea999b67d9446afb8"
format_stamp: "Formatted at 2026-09-29 21:04:52 on dist-test-slave-1dms"
server_key: "aa28d69ade4a35af2a5d7f15fe867969"
server_key_iv: "4ce8e2890e4bab1469f632c27c688d23"
server_key_version: "encryptionkey@0"
I20260929 21:04:52.736735 21969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "0e8efd24384f42bea999b67d9446afb8"
format_stamp: "Formatted at 2026-09-29 21:04:52 on dist-test-slave-1dms"
server_key: "aa28d69ade4a35af2a5d7f15fe867969"
server_key_iv: "4ce8e2890e4bab1469f632c27c688d23"
server_key_version: "encryptionkey@0"
I20260929 21:04:52.741652 21969 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 21:04:52.744976 21984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:52.746289 21969 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 21:04:52.746475 21969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "0e8efd24384f42bea999b67d9446afb8"
format_stamp: "Formatted at 2026-09-29 21:04:52 on dist-test-slave-1dms"
server_key: "aa28d69ade4a35af2a5d7f15fe867969"
server_key_iv: "4ce8e2890e4bab1469f632c27c688d23"
server_key_version: "encryptionkey@0"
I20260929 21:04:52.746685 21969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:52.752316 21969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:52.756177 21969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:52.756471 21969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:52.774549 21969 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38761
I20260929 21:04:52.774591 22035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38761 every 8 connection(s)
I20260929 21:04:52.775909 21969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:04:52.778867 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 21969
I20260929 21:04:52.779072 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:04:52.779522 388 external_mini_cluster.cc:1428] Setting key 8002fcb0f4601f850077553fd4ac5343
I20260929 21:04:52.779920 22036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:04:52.788375 22039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:52.781437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:54181 (local address 127.0.0.1:38761)
0929 21:04:52.782230 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:52.782246 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:52.782296 (+ 50us) server_negotiation.cc:407] Connection header received
0929 21:04:52.784597 (+ 2301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:52.784635 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:52.785103 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:52.785527 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:52.786502 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:52.786511 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:52.786532 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:52.786548 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:52.786636 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:52.786673 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:52.786886 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:52.787019 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 21:04:52.787212 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":322,"threads_started":1}
I20260929 21:04:52.792224 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 21:04:52.794142 22036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e8efd24384f42bea999b67d9446afb8: Bootstrap starting.
I20260929 21:04:52.797328 22036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e8efd24384f42bea999b67d9446afb8: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:52.798522 22036 log.cc:824] T 00000000000000000000000000000000 P 0e8efd24384f42bea999b67d9446afb8: Log is configured to *not* fsync() on all Append() calls
I20260929 21:04:52.799708 22039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:52.796929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:42739 (local address 127.0.0.1:38761)
0929 21:04:52.797097 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:52.797102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:52.797612 (+ 510us) server_negotiation.cc:407] Connection header received
0929 21:04:52.797759 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:52.797767 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:52.797837 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:52.797953 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:52.798943 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:52.798950 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:52.798956 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:52.798980 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:52.799044 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:52.799071 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:52.799200 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:52.799329 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 21:04:52.799376 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 21:04:52.801565 22036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e8efd24384f42bea999b67d9446afb8: No bootstrap required, opened a new log
I20260929 21:04:52.808641 22036 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e8efd24384f42bea999b67d9446afb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e8efd24384f42bea999b67d9446afb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38761 } }
I20260929 21:04:52.809054 22036 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e8efd24384f42bea999b67d9446afb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:52.809163 22036 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e8efd24384f42bea999b67d9446afb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e8efd24384f42bea999b67d9446afb8, State: Initialized, Role: FOLLOWER
I20260929 21:04:52.809830 22036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e8efd24384f42bea999b67d9446afb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e8efd24384f42bea999b67d9446afb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38761 } }
I20260929 21:04:52.810134 22036 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e8efd24384f42bea999b67d9446afb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:04:52.810292 22036 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e8efd24384f42bea999b67d9446afb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:04:52.810457 22036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e8efd24384f42bea999b67d9446afb8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:52.812551 22036 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e8efd24384f42bea999b67d9446afb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e8efd24384f42bea999b67d9446afb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38761 } }
I20260929 21:04:52.813071 22036 leader_election.cc:302] T 00000000000000000000000000000000 P 0e8efd24384f42bea999b67d9446afb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e8efd24384f42bea999b67d9446afb8; no voters:
I20260929 21:04:52.813617 22036 leader_election.cc:288] T 00000000000000000000000000000000 P 0e8efd24384f42bea999b67d9446afb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 21:04:52.813722 22046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e8efd24384f42bea999b67d9446afb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:04:52.814390 22046 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e8efd24384f42bea999b67d9446afb8 [term 1 LEADER]: Becoming Leader. State: Replica: 0e8efd24384f42bea999b67d9446afb8, State: Running, Role: LEADER
I20260929 21:04:52.814968 22046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e8efd24384f42bea999b67d9446afb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e8efd24384f42bea999b67d9446afb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38761 } }
I20260929 21:04:52.815594 22036 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e8efd24384f42bea999b67d9446afb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:04:52.817759 22047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e8efd24384f42bea999b67d9446afb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e8efd24384f42bea999b67d9446afb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e8efd24384f42bea999b67d9446afb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38761 } } }
I20260929 21:04:52.818563 22047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e8efd24384f42bea999b67d9446afb8 [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:52.818197 22048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e8efd24384f42bea999b67d9446afb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e8efd24384f42bea999b67d9446afb8. Latest consensus state: current_term: 1 leader_uuid: "0e8efd24384f42bea999b67d9446afb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e8efd24384f42bea999b67d9446afb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38761 } } }
I20260929 21:04:52.819325 22048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e8efd24384f42bea999b67d9446afb8 [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:52.826306 22055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:04:52.831733 22055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:04:52.840337 22055 catalog_manager.cc:1347] Generated new cluster ID: cba371ebb72546d2a6810829c3903031
I20260929 21:04:52.840487 22055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:04:52.849922 22055 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:04:52.850404 22055 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:04:52.858144 22055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e8efd24384f42bea999b67d9446afb8: Generated new TSK 0
I20260929 21:04:52.858963 22055 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 21:04:52.866815 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 21969
W20260929 21:04:52.867815 22031 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.108:54181
W20260929 21:04:52.868502 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 21969: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 21:04:52.868664 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 21969
2026-09-29T21:04:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1875 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 21:04:52 dist-test-slave-1dms krb5kdc[22067](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 21:04:52 dist-test-slave-1dms krb5kdc[22067](info): set up 2 sockets
Sep 29 21:04:52 dist-test-slave-1dms krb5kdc[22067](info): commencing operation
krb5kdc: starting...
W20260929 21:04:53.961217 388 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 21:04:54 dist-test-slave-1dms krb5kdc[22067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T21:04:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:04:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 21:04:54.211926 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42383
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:04:54.549578 22080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:54.549852 22080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:54.549943 22080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:54.557116 22080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:04:54.557221 22080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:54.557273 22080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:54.557336 22080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:04:54.557380 22080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:04:54.566887 22080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42383
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:54.568060 22080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:54.569352 22080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:54.580801 22085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:54.580900 22086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:54.580772 22088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:54.586639 22080 server_base.cc:1034] running on GCE node
I20260929 21:04:54.587455 22080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:54.588920 22080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:54.590179 22080 hybrid_clock.cc:648] HybridClock initialized: now 1790715894590134 us; error 57 us; skew 500 ppm
I20260929 21:04:54.590565 22080 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:04:54 dist-test-slave-1dms krb5kdc[22067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:04:54.593730 22080 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 21:04:54.595165 22080 webserver.cc:460] Webserver started at http://127.0.97.62:37711/ using document root <none> and password file <none>
I20260929 21:04:54.595860 22080 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:54.596004 22080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:54.596306 22080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:54.599270 22080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "6113309bdee942d0abc4e5feefe4112b"
format_stamp: "Formatted at 2026-09-29 21:04:54 on dist-test-slave-1dms"
server_key: "2e9ead7f88b10bddf22d7a6539f26c56"
server_key_iv: "3f1ed0a861cfffbbd8866562ddc7f539"
server_key_version: "encryptionkey@0"
I20260929 21:04:54.600050 22080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "6113309bdee942d0abc4e5feefe4112b"
format_stamp: "Formatted at 2026-09-29 21:04:54 on dist-test-slave-1dms"
server_key: "2e9ead7f88b10bddf22d7a6539f26c56"
server_key_iv: "3f1ed0a861cfffbbd8866562ddc7f539"
server_key_version: "encryptionkey@0"
I20260929 21:04:54.605075 22080 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260929 21:04:54.608353 22095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:54.609678 22080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:04:54.609862 22080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "6113309bdee942d0abc4e5feefe4112b"
format_stamp: "Formatted at 2026-09-29 21:04:54 on dist-test-slave-1dms"
server_key: "2e9ead7f88b10bddf22d7a6539f26c56"
server_key_iv: "3f1ed0a861cfffbbd8866562ddc7f539"
server_key_version: "encryptionkey@0"
I20260929 21:04:54.610049 22080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:54.638420 22080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:54.642858 22080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:54.643150 22080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:54.661072 22080 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42383
I20260929 21:04:54.661095 22146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42383 every 8 connection(s)
I20260929 21:04:54.662400 22080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:04:54.664436 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 22080
I20260929 21:04:54.664613 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:04:54.665133 388 external_mini_cluster.cc:1428] Setting key 04b48755a29b21f7d807504f13d8467c
I20260929 21:04:54.666494 22147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 21:04:54 dist-test-slave-1dms krb5kdc[22067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:04:54.680001 22147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6113309bdee942d0abc4e5feefe4112b: Bootstrap starting.
I20260929 21:04:54.684091 22147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6113309bdee942d0abc4e5feefe4112b: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:54.685449 22147 log.cc:824] T 00000000000000000000000000000000 P 6113309bdee942d0abc4e5feefe4112b: Log is configured to *not* fsync() on all Append() calls
I20260929 21:04:54.688192 22147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6113309bdee942d0abc4e5feefe4112b: No bootstrap required, opened a new log
I20260929 21:04:54.689431 22150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:54.666853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45467 (local address 127.0.97.62:42383)
0929 21:04:54.667593 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:54.667607 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:54.667655 (+ 48us) server_negotiation.cc:407] Connection header received
0929 21:04:54.669722 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:54.669747 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:54.670262 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:54.670702 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:54.672000 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:54.674045 (+ 2045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:54.675471 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:54.676495 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:54.676679 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:54.681183 (+ 4504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:54.681194 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:54.681222 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:04:54.681266 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:54.685309 (+ 4043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:54.686120 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:54.686130 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:54.686136 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:54.686310 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:54.686782 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:54.686792 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:54.686794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:54.687294 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:54.687639 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:54.688101 (+ 462us) server_negotiation.cc:299] Negotiation successful
0929 21:04:54.688412 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":299,"threads_started":1}
I20260929 21:04:54.693536 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 21:04:54.697024 22147 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6113309bdee942d0abc4e5feefe4112b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6113309bdee942d0abc4e5feefe4112b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42383 } }
I20260929 21:04:54.697773 22147 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6113309bdee942d0abc4e5feefe4112b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:54.697975 22147 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6113309bdee942d0abc4e5feefe4112b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6113309bdee942d0abc4e5feefe4112b, State: Initialized, Role: FOLLOWER
I20260929 21:04:54.698585 22147 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6113309bdee942d0abc4e5feefe4112b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6113309bdee942d0abc4e5feefe4112b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42383 } }
I20260929 21:04:54.698830 22147 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6113309bdee942d0abc4e5feefe4112b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:04:54.698988 22147 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6113309bdee942d0abc4e5feefe4112b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:04:54.699159 22147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6113309bdee942d0abc4e5feefe4112b [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:54.701531 22147 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6113309bdee942d0abc4e5feefe4112b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6113309bdee942d0abc4e5feefe4112b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42383 } }
I20260929 21:04:54.702010 22147 leader_election.cc:302] T 00000000000000000000000000000000 P 6113309bdee942d0abc4e5feefe4112b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6113309bdee942d0abc4e5feefe4112b; no voters:
I20260929 21:04:54.702664 22147 leader_election.cc:288] T 00000000000000000000000000000000 P 6113309bdee942d0abc4e5feefe4112b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 21:04:54.702785 22157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6113309bdee942d0abc4e5feefe4112b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:04:54.704695 22157 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6113309bdee942d0abc4e5feefe4112b [term 1 LEADER]: Becoming Leader. State: Replica: 6113309bdee942d0abc4e5feefe4112b, State: Running, Role: LEADER
I20260929 21:04:54.705199 22147 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6113309bdee942d0abc4e5feefe4112b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:04:54.705327 22157 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6113309bdee942d0abc4e5feefe4112b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6113309bdee942d0abc4e5feefe4112b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42383 } }
I20260929 21:04:54.709322 22158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6113309bdee942d0abc4e5feefe4112b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6113309bdee942d0abc4e5feefe4112b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6113309bdee942d0abc4e5feefe4112b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42383 } } }
I20260929 21:04:54.709805 22158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6113309bdee942d0abc4e5feefe4112b [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:54.709985 22159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6113309bdee942d0abc4e5feefe4112b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6113309bdee942d0abc4e5feefe4112b. Latest consensus state: current_term: 1 leader_uuid: "6113309bdee942d0abc4e5feefe4112b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6113309bdee942d0abc4e5feefe4112b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42383 } } }
I20260929 21:04:54.710399 22159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6113309bdee942d0abc4e5feefe4112b [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:54.711956 22166 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:04:54.717231 22150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:54.697676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57847 (local address 127.0.97.62:42383)
0929 21:04:54.697880 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:54.697887 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:54.698299 (+ 412us) server_negotiation.cc:407] Connection header received
0929 21:04:54.698485 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:54.698494 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:54.698549 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:54.698688 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:54.700207 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:54.701032 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:54.702380 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:54.703001 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:54.703160 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:54.706228 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:54.706238 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:54.706241 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:04:54.706269 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:54.713105 (+ 6836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:54.714278 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:54.714288 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:54.714294 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:54.714571 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:04:54.715142 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:04:54.715150 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:04:54.715164 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:04:54.715634 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:54.716016 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:54.716450 (+ 434us) server_negotiation.cc:299] Negotiation successful
0929 21:04:54.716690 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 21:04:54.721982 22166 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:04:54.732223 22166 catalog_manager.cc:1347] Generated new cluster ID: 9d8dfec4db204390a481f00aa96dc5d2
I20260929 21:04:54.732378 22166 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:04:54.753113 22166 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:04:54.754395 22166 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:04:54.769644 22166 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6113309bdee942d0abc4e5feefe4112b: Generated new TSK 0
I20260929 21:04:54.770448 22166 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 21:04:54.776405 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 22080
I20260929 21:04:54.777716 22142 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:04:54.868528 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 22080
W20260929 21:04:54.885424 22065 connection.cc:535] client connection to 127.0.97.62:42383 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:42383): Connection reset by peer (error 104)
2026-09-29T21:04:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2003 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 21:04:54 dist-test-slave-1dms krb5kdc[22179](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 21:04:54 dist-test-slave-1dms krb5kdc[22179](info): set up 2 sockets
Sep 29 21:04:54 dist-test-slave-1dms krb5kdc[22179](info): commencing operation
krb5kdc: starting...
W20260929 21:04:55.969182 388 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.008s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 21:04:56 dist-test-slave-1dms krb5kdc[22179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T21:04:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:04:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 21:04:56.228682 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38667
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:04:56.556306 22192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:56.556583 22192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:56.556785 22192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:56.563990 22192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:04:56.564090 22192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:56.564193 22192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:56.564303 22192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:04:56.564381 22192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:04:56.574328 22192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38667
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:56.575655 22192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:56.577040 22192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:56.588549 22198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:56.588704 22200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:56.588498 22197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:56.589298 22192 server_base.cc:1034] running on GCE node
I20260929 21:04:56.590013 22192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:56.591373 22192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:56.592626 22192 hybrid_clock.cc:648] HybridClock initialized: now 1790715896592578 us; error 54 us; skew 500 ppm
I20260929 21:04:56.593068 22192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:04:56 dist-test-slave-1dms krb5kdc[22179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:04:56.596139 22192 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 21:04:56.597579 22192 webserver.cc:460] Webserver started at http://127.0.97.62:33779/ using document root <none> and password file <none>
I20260929 21:04:56.598191 22192 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:56.598299 22192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:56.598544 22192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:56.601424 22192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "228fa71eb57941c5a2dcc7fa1d61d8bb"
format_stamp: "Formatted at 2026-09-29 21:04:56 on dist-test-slave-1dms"
server_key: "8856ae69d0276e5710a1b5506e29fe2f"
server_key_iv: "b3dcdd905e4854ec360889033b3cb217"
server_key_version: "encryptionkey@0"
I20260929 21:04:56.602106 22192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "228fa71eb57941c5a2dcc7fa1d61d8bb"
format_stamp: "Formatted at 2026-09-29 21:04:56 on dist-test-slave-1dms"
server_key: "8856ae69d0276e5710a1b5506e29fe2f"
server_key_iv: "b3dcdd905e4854ec360889033b3cb217"
server_key_version: "encryptionkey@0"
I20260929 21:04:56.607028 22192 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 21:04:56.610280 22207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:56.611527 22192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 21:04:56.611658 22192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "228fa71eb57941c5a2dcc7fa1d61d8bb"
format_stamp: "Formatted at 2026-09-29 21:04:56 on dist-test-slave-1dms"
server_key: "8856ae69d0276e5710a1b5506e29fe2f"
server_key_iv: "b3dcdd905e4854ec360889033b3cb217"
server_key_version: "encryptionkey@0"
I20260929 21:04:56.611811 22192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:56.655299 22192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:56.659243 22192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:56.659478 22192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:56.677902 22192 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38667
I20260929 21:04:56.677944 22258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38667 every 8 connection(s)
I20260929 21:04:56.679232 22192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:04:56.682915 22259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:04:56.682992 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 22192
I20260929 21:04:56.683151 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:04:56.683678 388 external_mini_cluster.cc:1428] Setting key a27c8443fa0d447d3a8b9f7a4403d405
I20260929 21:04:56.695920 22259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 228fa71eb57941c5a2dcc7fa1d61d8bb: Bootstrap starting.
I20260929 21:04:56.697113 22262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:56.685418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48751 (local address 127.0.97.62:38667)
0929 21:04:56.686266 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:56.686277 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:56.686320 (+ 43us) server_negotiation.cc:407] Connection header received
0929 21:04:56.687889 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:56.687910 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:56.688450 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:56.689616 (+ 1166us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:56.690005 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:56.691274 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:56.692590 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:56.693351 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:56.693570 (+ 219us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:56.694843 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:56.694851 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:56.694863 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:56.695384 (+ 521us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:56.695455 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:56.695478 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:56.695661 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:56.695940 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 21:04:56.696090 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":434,"threads_started":1}
I20260929 21:04:56.699337 22259 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 228fa71eb57941c5a2dcc7fa1d61d8bb: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:56.700469 22259 log.cc:824] T 00000000000000000000000000000000 P 228fa71eb57941c5a2dcc7fa1d61d8bb: Log is configured to *not* fsync() on all Append() calls
I20260929 21:04:56.700765 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 21:04:56.703133 22259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 228fa71eb57941c5a2dcc7fa1d61d8bb: No bootstrap required, opened a new log
I20260929 21:04:56.712183 22262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:56.705097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42901 (local address 127.0.97.62:38667)
0929 21:04:56.705281 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:56.705289 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:56.705839 (+ 550us) server_negotiation.cc:407] Connection header received
0929 21:04:56.706004 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:56.706008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:56.706061 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:56.706144 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:56.706717 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:56.707469 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:56.709526 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:04:56.709993 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:04:56.710096 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:04:56.711337 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:56.711343 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:56.711355 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:56.711384 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:56.711437 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:56.711456 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:56.711549 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:56.711780 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 21:04:56.711847 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 21:04:56.713177 22259 raft_consensus.cc:348] T 00000000000000000000000000000000 P 228fa71eb57941c5a2dcc7fa1d61d8bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "228fa71eb57941c5a2dcc7fa1d61d8bb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38667 } }
I20260929 21:04:56.713546 22259 raft_consensus.cc:374] T 00000000000000000000000000000000 P 228fa71eb57941c5a2dcc7fa1d61d8bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:56.713666 22259 raft_consensus.cc:729] T 00000000000000000000000000000000 P 228fa71eb57941c5a2dcc7fa1d61d8bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 228fa71eb57941c5a2dcc7fa1d61d8bb, State: Initialized, Role: FOLLOWER
I20260929 21:04:56.714285 22259 consensus_queue.cc:260] T 00000000000000000000000000000000 P 228fa71eb57941c5a2dcc7fa1d61d8bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "228fa71eb57941c5a2dcc7fa1d61d8bb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38667 } }
I20260929 21:04:56.714519 22259 raft_consensus.cc:388] T 00000000000000000000000000000000 P 228fa71eb57941c5a2dcc7fa1d61d8bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:04:56.714681 22259 raft_consensus.cc:482] T 00000000000000000000000000000000 P 228fa71eb57941c5a2dcc7fa1d61d8bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:04:56.714866 22259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 228fa71eb57941c5a2dcc7fa1d61d8bb [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:56.717197 22259 raft_consensus.cc:504] T 00000000000000000000000000000000 P 228fa71eb57941c5a2dcc7fa1d61d8bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "228fa71eb57941c5a2dcc7fa1d61d8bb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38667 } }
I20260929 21:04:56.717669 22259 leader_election.cc:302] T 00000000000000000000000000000000 P 228fa71eb57941c5a2dcc7fa1d61d8bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 228fa71eb57941c5a2dcc7fa1d61d8bb; no voters:
I20260929 21:04:56.718297 22259 leader_election.cc:288] T 00000000000000000000000000000000 P 228fa71eb57941c5a2dcc7fa1d61d8bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 21:04:56.718462 22269 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 228fa71eb57941c5a2dcc7fa1d61d8bb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:04:56.719470 22269 raft_consensus.cc:686] T 00000000000000000000000000000000 P 228fa71eb57941c5a2dcc7fa1d61d8bb [term 1 LEADER]: Becoming Leader. State: Replica: 228fa71eb57941c5a2dcc7fa1d61d8bb, State: Running, Role: LEADER
I20260929 21:04:56.720000 22269 consensus_queue.cc:237] T 00000000000000000000000000000000 P 228fa71eb57941c5a2dcc7fa1d61d8bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "228fa71eb57941c5a2dcc7fa1d61d8bb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38667 } }
I20260929 21:04:56.720346 22259 sys_catalog.cc:564] T 00000000000000000000000000000000 P 228fa71eb57941c5a2dcc7fa1d61d8bb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:04:56.722803 22271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 228fa71eb57941c5a2dcc7fa1d61d8bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 228fa71eb57941c5a2dcc7fa1d61d8bb. Latest consensus state: current_term: 1 leader_uuid: "228fa71eb57941c5a2dcc7fa1d61d8bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "228fa71eb57941c5a2dcc7fa1d61d8bb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38667 } } }
I20260929 21:04:56.723179 22271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 228fa71eb57941c5a2dcc7fa1d61d8bb [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:56.723786 22270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 228fa71eb57941c5a2dcc7fa1d61d8bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "228fa71eb57941c5a2dcc7fa1d61d8bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "228fa71eb57941c5a2dcc7fa1d61d8bb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38667 } } }
I20260929 21:04:56.724076 22270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 228fa71eb57941c5a2dcc7fa1d61d8bb [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:56.725713 22278 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:04:56.730862 22278 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:04:56.739826 22278 catalog_manager.cc:1347] Generated new cluster ID: 25bbf2a222c54695b8423723ac3001a2
I20260929 21:04:56.739938 22278 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:04:56.753533 22278 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:04:56.755348 22278 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:04:56.764348 22278 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 228fa71eb57941c5a2dcc7fa1d61d8bb: Generated new TSK 0
I20260929 21:04:56.765404 22278 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 21:04:56.782562 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 22192
W20260929 21:04:56.783742 22254 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48751
W20260929 21:04:56.784482 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 22192: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 21:04:56.784639 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 22192
W20260929 21:04:56.801729 22177 connection.cc:535] client connection to 127.0.97.62:38667 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:38667): Connection reset by peer (error 104)
2026-09-29T21:04:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1915 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 21:04:56 dist-test-slave-1dms krb5kdc[22290](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 21:04:56 dist-test-slave-1dms krb5kdc[22290](info): set up 2 sockets
Sep 29 21:04:56 dist-test-slave-1dms krb5kdc[22290](info): commencing operation
krb5kdc: starting...
W20260929 21:04:57.873286 388 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 21:04:58 dist-test-slave-1dms krb5kdc[22290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T21:04:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:04:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 21:04:58.116513 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39883
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:04:58.441208 22303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:04:58.441587 22303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:04:58.441759 22303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:04:58.449220 22303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:04:58.449383 22303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:04:58.449487 22303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:04:58.449599 22303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:04:58.449679 22303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:04:58.459645 22303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39883
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:04:58.461042 22303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:04:58.462335 22303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:04:58.474289 22309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:58.474267 22311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:04:58.474239 22308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:04:58.475059 22303 server_base.cc:1034] running on GCE node
I20260929 21:04:58.475775 22303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:04:58.477207 22303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:04:58.478407 22303 hybrid_clock.cc:648] HybridClock initialized: now 1790715898478371 us; error 31 us; skew 500 ppm
I20260929 21:04:58.478746 22303 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:04:58 dist-test-slave-1dms krb5kdc[22290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:04:58.489084 22303 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 21:04:58.490536 22303 webserver.cc:460] Webserver started at http://127.0.97.62:39661/ using document root <none> and password file <none>
I20260929 21:04:58.491230 22303 fs_manager.cc:359] Metadata directory not provided
I20260929 21:04:58.491372 22303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:04:58.491676 22303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:04:58.494732 22303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "f8615134004d43e6a2e1182d6253acf0"
format_stamp: "Formatted at 2026-09-29 21:04:58 on dist-test-slave-1dms"
server_key: "4fa5730df17e6f94c70200d45b4a8ff1"
server_key_iv: "262bc70de80877b837499c046c9168ea"
server_key_version: "encryptionkey@0"
I20260929 21:04:58.495476 22303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "f8615134004d43e6a2e1182d6253acf0"
format_stamp: "Formatted at 2026-09-29 21:04:58 on dist-test-slave-1dms"
server_key: "4fa5730df17e6f94c70200d45b4a8ff1"
server_key_iv: "262bc70de80877b837499c046c9168ea"
server_key_version: "encryptionkey@0"
I20260929 21:04:58.500371 22303 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 21:04:58.504067 22318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:04:58.505952 22303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:04:58.506189 22303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "f8615134004d43e6a2e1182d6253acf0"
format_stamp: "Formatted at 2026-09-29 21:04:58 on dist-test-slave-1dms"
server_key: "4fa5730df17e6f94c70200d45b4a8ff1"
server_key_iv: "262bc70de80877b837499c046c9168ea"
server_key_version: "encryptionkey@0"
I20260929 21:04:58.506510 22303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:04:58.513916 22303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:04:58.517868 22303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:04:58.518146 22303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:04:58.536062 22303 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39883
I20260929 21:04:58.536105 22369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39883 every 8 connection(s)
I20260929 21:04:58.537565 22303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:04:58.539016 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 22303
I20260929 21:04:58.539192 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:04:58.539752 388 external_mini_cluster.cc:1428] Setting key 658f5927db5445beed282afe7160a5db
I20260929 21:04:58.542150 22370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:04:58.547271 22373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:58.541540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39661 (local address 127.0.97.62:39883)
0929 21:04:58.542336 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:58.542353 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:58.542394 (+ 41us) server_negotiation.cc:407] Connection header received
0929 21:04:58.543951 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:58.543972 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:58.544293 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:58.544636 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:58.545706 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:58.545712 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:58.545724 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:58.545737 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:58.545811 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:58.545830 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:58.545989 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:58.546123 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 21:04:58.546318 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":342,"threads_started":1}
I20260929 21:04:58.550918 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 21:04:58.554459 22370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8615134004d43e6a2e1182d6253acf0: Bootstrap starting.
I20260929 21:04:58.558449 22370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8615134004d43e6a2e1182d6253acf0: Neither blocks nor log segments found. Creating new log.
I20260929 21:04:58.559037 22373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:04:58.556207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53981 (local address 127.0.97.62:39883)
0929 21:04:58.556450 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 21:04:58.556461 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 21:04:58.556784 (+ 323us) server_negotiation.cc:407] Connection header received
0929 21:04:58.557008 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:04:58.557017 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:04:58.557084 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:04:58.557203 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:04:58.558236 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:04:58.558241 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:04:58.558244 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:04:58.558270 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:04:58.558312 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:04:58.558332 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:04:58.558441 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 21:04:58.558611 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 21:04:58.558681 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 21:04:58.559717 22370 log.cc:824] T 00000000000000000000000000000000 P f8615134004d43e6a2e1182d6253acf0: Log is configured to *not* fsync() on all Append() calls
I20260929 21:04:58.562453 22370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8615134004d43e6a2e1182d6253acf0: No bootstrap required, opened a new log
I20260929 21:04:58.570245 22370 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8615134004d43e6a2e1182d6253acf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8615134004d43e6a2e1182d6253acf0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39883 } }
I20260929 21:04:58.570582 22370 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8615134004d43e6a2e1182d6253acf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:04:58.570653 22370 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8615134004d43e6a2e1182d6253acf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8615134004d43e6a2e1182d6253acf0, State: Initialized, Role: FOLLOWER
I20260929 21:04:58.571142 22370 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8615134004d43e6a2e1182d6253acf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8615134004d43e6a2e1182d6253acf0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39883 } }
I20260929 21:04:58.571332 22370 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8615134004d43e6a2e1182d6253acf0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:04:58.571429 22370 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8615134004d43e6a2e1182d6253acf0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:04:58.571549 22370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8615134004d43e6a2e1182d6253acf0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:04:58.574033 22370 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8615134004d43e6a2e1182d6253acf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8615134004d43e6a2e1182d6253acf0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39883 } }
I20260929 21:04:58.574535 22370 leader_election.cc:302] T 00000000000000000000000000000000 P f8615134004d43e6a2e1182d6253acf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8615134004d43e6a2e1182d6253acf0; no voters:
I20260929 21:04:58.575100 22370 leader_election.cc:288] T 00000000000000000000000000000000 P f8615134004d43e6a2e1182d6253acf0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 21:04:58.575246 22380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8615134004d43e6a2e1182d6253acf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:04:58.575917 22380 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8615134004d43e6a2e1182d6253acf0 [term 1 LEADER]: Becoming Leader. State: Replica: f8615134004d43e6a2e1182d6253acf0, State: Running, Role: LEADER
I20260929 21:04:58.576409 22380 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8615134004d43e6a2e1182d6253acf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8615134004d43e6a2e1182d6253acf0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39883 } }
I20260929 21:04:58.577181 22370 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8615134004d43e6a2e1182d6253acf0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:04:58.579238 22382 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8615134004d43e6a2e1182d6253acf0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8615134004d43e6a2e1182d6253acf0. Latest consensus state: current_term: 1 leader_uuid: "f8615134004d43e6a2e1182d6253acf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8615134004d43e6a2e1182d6253acf0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39883 } } }
I20260929 21:04:58.579681 22382 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8615134004d43e6a2e1182d6253acf0 [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:58.579253 22381 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8615134004d43e6a2e1182d6253acf0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8615134004d43e6a2e1182d6253acf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8615134004d43e6a2e1182d6253acf0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39883 } } }
I20260929 21:04:58.582023 22381 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8615134004d43e6a2e1182d6253acf0 [sys.catalog]: This master's current role is: LEADER
I20260929 21:04:58.584424 22387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:04:58.591111 22387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:04:58.600164 22387 catalog_manager.cc:1347] Generated new cluster ID: f30f9443c49c467d9e7c4479d1da4889
I20260929 21:04:58.600287 22387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:04:58.625249 22387 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:04:58.625674 22387 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:04:58.637830 22387 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8615134004d43e6a2e1182d6253acf0: Generated new TSK 0
I20260929 21:04:58.638667 22387 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 21:04:58.648972 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 22303
W20260929 21:04:58.650029 22365 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39661
W20260929 21:04:58.650869 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 22303: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 21:04:58.651022 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 22303
2026-09-29T21:04:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1865 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 21:04:58 dist-test-slave-1dms krb5kdc[22401](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 21:04:58 dist-test-slave-1dms krb5kdc[22401](info): set up 2 sockets
Sep 29 21:04:58 dist-test-slave-1dms krb5kdc[22401](info): commencing operation
krb5kdc: starting...
W20260929 21:04:59.761188 388 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 21:04:59 dist-test-slave-1dms krb5kdc[22401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T21:04:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:04:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 21:05:00.012593 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36333
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:05:00.341109 22414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:05:00.341390 22414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:05:00.341562 22414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:05:00.348639 22414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:05:00.348742 22414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:05:00.348788 22414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:05:00.348892 22414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:05:00.348986 22414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:05:00.359170 22414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:05:00.360504 22414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:05:00.361887 22414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:05:00.374401 22420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:05:00.374497 22419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:05:00.374460 22422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:05:00.375398 22414 server_base.cc:1034] running on GCE node
I20260929 21:05:00.376102 22414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:05:00.377506 22414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:05:00.378737 22414 hybrid_clock.cc:648] HybridClock initialized: now 1790715900378700 us; error 47 us; skew 500 ppm
I20260929 21:05:00.379093 22414 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:05:00 dist-test-slave-1dms krb5kdc[22401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:05:00.382256 22414 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 21:05:00.383621 22414 webserver.cc:460] Webserver started at http://127.0.97.62:37105/ using document root <none> and password file <none>
I20260929 21:05:00.384253 22414 fs_manager.cc:359] Metadata directory not provided
I20260929 21:05:00.384346 22414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:05:00.384586 22414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:05:00.387523 22414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "6743ae6abdab4064a7e8258a186d938a"
format_stamp: "Formatted at 2026-09-29 21:05:00 on dist-test-slave-1dms"
server_key: "b0c208f2f076b873a8ab1732ca468974"
server_key_iv: "f054116e143ee7978a3cc225bece56f8"
server_key_version: "encryptionkey@0"
I20260929 21:05:00.388214 22414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "6743ae6abdab4064a7e8258a186d938a"
format_stamp: "Formatted at 2026-09-29 21:05:00 on dist-test-slave-1dms"
server_key: "b0c208f2f076b873a8ab1732ca468974"
server_key_iv: "f054116e143ee7978a3cc225bece56f8"
server_key_version: "encryptionkey@0"
I20260929 21:05:00.393082 22414 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 21:05:00.396284 22429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:05:00.397570 22414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:05:00.397706 22414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "6743ae6abdab4064a7e8258a186d938a"
format_stamp: "Formatted at 2026-09-29 21:05:00 on dist-test-slave-1dms"
server_key: "b0c208f2f076b873a8ab1732ca468974"
server_key_iv: "f054116e143ee7978a3cc225bece56f8"
server_key_version: "encryptionkey@0"
I20260929 21:05:00.397866 22414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:05:00.426525 22414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:05:00.431130 22414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:05:00.431474 22414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:05:00.450306 22414 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36333
I20260929 21:05:00.450336 22480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36333 every 8 connection(s)
I20260929 21:05:00.451634 22414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:05:00.455212 22481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:05:00.457564 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 22414
I20260929 21:05:00.457736 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:05:00.458256 388 external_mini_cluster.cc:1428] Setting key 9ae822d8da5c925982813d18e06ca35e
I20260929 21:05:00.467523 22481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6743ae6abdab4064a7e8258a186d938a: Bootstrap starting.
I20260929 21:05:00.471513 22481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6743ae6abdab4064a7e8258a186d938a: Neither blocks nor log segments found. Creating new log.
Sep 29 21:05:00 dist-test-slave-1dms krb5kdc[22401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:05:00.472734 22481 log.cc:824] T 00000000000000000000000000000000 P 6743ae6abdab4064a7e8258a186d938a: Log is configured to *not* fsync() on all Append() calls
I20260929 21:05:00.476051 22481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6743ae6abdab4064a7e8258a186d938a: No bootstrap required, opened a new log
I20260929 21:05:00.481554 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:00.459991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47251 (local address 127.0.97.62:36333)
0929 21:05:00.460782 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:00.460793 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:00.460884 (+ 91us) server_negotiation.cc:407] Connection header received
0929 21:05:00.462537 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:00.462559 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:00.462952 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:00.463266 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:00.464976 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:00.466287 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:00.468001 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:00.468725 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:00.468924 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:00.473701 (+ 4777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:05:00.473715 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:05:00.473736 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:05:00.473792 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:05:00.477336 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:00.478182 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:00.478202 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:00.478210 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:00.478399 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:00.478836 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:00.478840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:00.478842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:00.479380 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:05:00.479603 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:00.480067 (+ 464us) server_negotiation.cc:299] Negotiation successful
0929 21:05:00.480459 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":363,"threads_started":1}
I20260929 21:05:00.485515 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 21:05:00.485442 22481 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6743ae6abdab4064a7e8258a186d938a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6743ae6abdab4064a7e8258a186d938a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36333 } }
I20260929 21:05:00.485877 22481 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6743ae6abdab4064a7e8258a186d938a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:05:00.485968 22481 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6743ae6abdab4064a7e8258a186d938a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6743ae6abdab4064a7e8258a186d938a, State: Initialized, Role: FOLLOWER
I20260929 21:05:00.486568 22481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6743ae6abdab4064a7e8258a186d938a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6743ae6abdab4064a7e8258a186d938a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36333 } }
I20260929 21:05:00.486871 22481 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6743ae6abdab4064a7e8258a186d938a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:05:00.486995 22481 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6743ae6abdab4064a7e8258a186d938a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:05:00.487149 22481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6743ae6abdab4064a7e8258a186d938a [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:05:00.490330 22481 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6743ae6abdab4064a7e8258a186d938a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6743ae6abdab4064a7e8258a186d938a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36333 } }
I20260929 21:05:00.490989 22481 leader_election.cc:302] T 00000000000000000000000000000000 P 6743ae6abdab4064a7e8258a186d938a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6743ae6abdab4064a7e8258a186d938a; no voters:
I20260929 21:05:00.491647 22481 leader_election.cc:288] T 00000000000000000000000000000000 P 6743ae6abdab4064a7e8258a186d938a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 21:05:00.491787 22491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6743ae6abdab4064a7e8258a186d938a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:05:00.493935 22481 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6743ae6abdab4064a7e8258a186d938a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:05:00.498601 22491 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6743ae6abdab4064a7e8258a186d938a [term 1 LEADER]: Becoming Leader. State: Replica: 6743ae6abdab4064a7e8258a186d938a, State: Running, Role: LEADER
I20260929 21:05:00.499239 22491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6743ae6abdab4064a7e8258a186d938a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6743ae6abdab4064a7e8258a186d938a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36333 } }
I20260929 21:05:00.502171 22492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6743ae6abdab4064a7e8258a186d938a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6743ae6abdab4064a7e8258a186d938a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6743ae6abdab4064a7e8258a186d938a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36333 } } }
I20260929 21:05:00.502840 22492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6743ae6abdab4064a7e8258a186d938a [sys.catalog]: This master's current role is: LEADER
I20260929 21:05:00.502755 22493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6743ae6abdab4064a7e8258a186d938a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6743ae6abdab4064a7e8258a186d938a. Latest consensus state: current_term: 1 leader_uuid: "6743ae6abdab4064a7e8258a186d938a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6743ae6abdab4064a7e8258a186d938a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36333 } } }
I20260929 21:05:00.503082 22493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6743ae6abdab4064a7e8258a186d938a [sys.catalog]: This master's current role is: LEADER
I20260929 21:05:00.511657 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:00.490603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58015 (local address 127.0.97.62:36333)
0929 21:05:00.490781 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:00.490790 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:00.491266 (+ 476us) server_negotiation.cc:407] Connection header received
0929 21:05:00.491483 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:00.491490 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:00.491583 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:00.491705 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:00.493617 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:00.495167 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:00.497610 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:00.498591 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:00.498760 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:00.502363 (+ 3603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:05:00.502376 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:05:00.502381 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:05:00.502419 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:05:00.506991 (+ 4572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:00.507983 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:00.507991 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:00.507996 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:00.508180 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:00.508696 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:00.508704 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:00.508710 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:00.509240 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:05:00.509532 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:00.510851 (+ 1319us) server_negotiation.cc:299] Negotiation successful
0929 21:05:00.511144 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 21:05:00.513266 22505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:05:00.518512 22505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:05:00.527508 22505 catalog_manager.cc:1347] Generated new cluster ID: d3f41b37789045379da5694503efac00
I20260929 21:05:00.527618 22505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:05:00.542095 22505 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:05:00.543921 22505 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:05:00.550242 22505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6743ae6abdab4064a7e8258a186d938a: Generated new TSK 0
I20260929 21:05:00.551210 22505 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 21:05:00.556569 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 22414
I20260929 21:05:00.557606 22476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:05:00.646592 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 22414
2026-09-29T21:05:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1995 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 21:05:00 dist-test-slave-1dms krb5kdc[22513](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 21:05:00 dist-test-slave-1dms krb5kdc[22513](info): set up 2 sockets
Sep 29 21:05:00 dist-test-slave-1dms krb5kdc[22513](info): commencing operation
krb5kdc: starting...
W20260929 21:05:01.741436 388 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 21:05:01 dist-test-slave-1dms krb5kdc[22513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T21:05:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:05:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 21:05:01.987561 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36805
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:05:02.314167 22526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:05:02.314528 22526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:05:02.314687 22526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:05:02.321976 22526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:05:02.322131 22526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:05:02.322243 22526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:05:02.322350 22526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:05:02.322427 22526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:05:02.332464 22526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:05:02.333841 22526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:05:02.335177 22526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:05:02.347088 22531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:05:02.347219 22532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:05:02.347832 22534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:05:02.348286 22526 server_base.cc:1034] running on GCE node
I20260929 21:05:02.349098 22526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:05:02.350521 22526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:05:02.351747 22526 hybrid_clock.cc:648] HybridClock initialized: now 1790715902351693 us; error 61 us; skew 500 ppm
I20260929 21:05:02.352084 22526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:05:02 dist-test-slave-1dms krb5kdc[22513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:05:02.355137 22526 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 21:05:02.356575 22526 webserver.cc:460] Webserver started at http://127.0.97.62:39239/ using document root <none> and password file <none>
I20260929 21:05:02.357300 22526 fs_manager.cc:359] Metadata directory not provided
I20260929 21:05:02.357453 22526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:05:02.357755 22526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:05:02.360606 22526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "c40e7cffb88d4c38b95f967c984fe80f"
format_stamp: "Formatted at 2026-09-29 21:05:02 on dist-test-slave-1dms"
server_key: "78b541cfcd90f3223aa372fb328f408a"
server_key_iv: "3e5efd6cf33477a6bd96352d2aa5f4bd"
server_key_version: "encryptionkey@0"
I20260929 21:05:02.361409 22526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "c40e7cffb88d4c38b95f967c984fe80f"
format_stamp: "Formatted at 2026-09-29 21:05:02 on dist-test-slave-1dms"
server_key: "78b541cfcd90f3223aa372fb328f408a"
server_key_iv: "3e5efd6cf33477a6bd96352d2aa5f4bd"
server_key_version: "encryptionkey@0"
I20260929 21:05:02.366324 22526 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 21:05:02.369732 22541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:05:02.371017 22526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 21:05:02.371189 22526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "c40e7cffb88d4c38b95f967c984fe80f"
format_stamp: "Formatted at 2026-09-29 21:05:02 on dist-test-slave-1dms"
server_key: "78b541cfcd90f3223aa372fb328f408a"
server_key_iv: "3e5efd6cf33477a6bd96352d2aa5f4bd"
server_key_version: "encryptionkey@0"
I20260929 21:05:02.371390 22526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:05:02.392131 22526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:05:02.395989 22526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:05:02.396281 22526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:05:02.415211 22526 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36805
I20260929 21:05:02.415230 22592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36805 every 8 connection(s)
I20260929 21:05:02.416607 22526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:05:02.420270 22593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:05:02.422267 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 22526
I20260929 21:05:02.422446 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:05:02.422993 388 external_mini_cluster.cc:1428] Setting key 529f6be5e7bad908108958d118a56aa0
I20260929 21:05:02.434273 22593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c40e7cffb88d4c38b95f967c984fe80f: Bootstrap starting.
I20260929 21:05:02.436364 22596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:02.424801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32997 (local address 127.0.97.62:36805)
0929 21:05:02.425793 (+ 992us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:02.425804 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:02.425844 (+ 40us) server_negotiation.cc:407] Connection header received
0929 21:05:02.427429 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:02.427452 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:02.427857 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:02.428218 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:02.428801 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:02.430141 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:02.432004 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:02.432668 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:02.432803 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:02.434087 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:05:02.434095 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:05:02.434107 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:05:02.434685 (+ 578us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:05:02.434770 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:05:02.434794 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:05:02.434880 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:02.435113 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 21:05:02.435420 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723,"thread_start_us":459,"threads_started":1}
I20260929 21:05:02.437549 22593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c40e7cffb88d4c38b95f967c984fe80f: Neither blocks nor log segments found. Creating new log.
I20260929 21:05:02.438597 22593 log.cc:824] T 00000000000000000000000000000000 P c40e7cffb88d4c38b95f967c984fe80f: Log is configured to *not* fsync() on all Append() calls
I20260929 21:05:02.440042 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 21:05:02.441464 22593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c40e7cffb88d4c38b95f967c984fe80f: No bootstrap required, opened a new log
I20260929 21:05:02.450626 22593 raft_consensus.cc:348] T 00000000000000000000000000000000 P c40e7cffb88d4c38b95f967c984fe80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c40e7cffb88d4c38b95f967c984fe80f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36805 } }
I20260929 21:05:02.451164 22593 raft_consensus.cc:374] T 00000000000000000000000000000000 P c40e7cffb88d4c38b95f967c984fe80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:05:02.451231 22596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:02.444576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37233 (local address 127.0.97.62:36805)
0929 21:05:02.444789 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:02.444795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:02.445163 (+ 368us) server_negotiation.cc:407] Connection header received
0929 21:05:02.445370 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:02.445373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:02.445438 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:02.445522 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:02.446138 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:02.446888 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:02.448734 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:02.449271 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:02.449376 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:02.450520 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:05:02.450524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:05:02.450527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:05:02.450543 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:05:02.450580 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:05:02.450594 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:05:02.450665 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:02.450826 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 21:05:02.450893 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 21:05:02.451316 22593 raft_consensus.cc:729] T 00000000000000000000000000000000 P c40e7cffb88d4c38b95f967c984fe80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c40e7cffb88d4c38b95f967c984fe80f, State: Initialized, Role: FOLLOWER
I20260929 21:05:02.451980 22593 consensus_queue.cc:260] T 00000000000000000000000000000000 P c40e7cffb88d4c38b95f967c984fe80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c40e7cffb88d4c38b95f967c984fe80f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36805 } }
I20260929 21:05:02.452235 22593 raft_consensus.cc:388] T 00000000000000000000000000000000 P c40e7cffb88d4c38b95f967c984fe80f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:05:02.452397 22593 raft_consensus.cc:482] T 00000000000000000000000000000000 P c40e7cffb88d4c38b95f967c984fe80f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:05:02.452582 22593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c40e7cffb88d4c38b95f967c984fe80f [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:05:02.454797 22593 raft_consensus.cc:504] T 00000000000000000000000000000000 P c40e7cffb88d4c38b95f967c984fe80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c40e7cffb88d4c38b95f967c984fe80f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36805 } }
I20260929 21:05:02.455288 22593 leader_election.cc:302] T 00000000000000000000000000000000 P c40e7cffb88d4c38b95f967c984fe80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c40e7cffb88d4c38b95f967c984fe80f; no voters:
I20260929 21:05:02.455830 22593 leader_election.cc:288] T 00000000000000000000000000000000 P c40e7cffb88d4c38b95f967c984fe80f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 21:05:02.455969 22603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c40e7cffb88d4c38b95f967c984fe80f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:05:02.456691 22603 raft_consensus.cc:686] T 00000000000000000000000000000000 P c40e7cffb88d4c38b95f967c984fe80f [term 1 LEADER]: Becoming Leader. State: Replica: c40e7cffb88d4c38b95f967c984fe80f, State: Running, Role: LEADER
I20260929 21:05:02.457237 22603 consensus_queue.cc:237] T 00000000000000000000000000000000 P c40e7cffb88d4c38b95f967c984fe80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c40e7cffb88d4c38b95f967c984fe80f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36805 } }
I20260929 21:05:02.458026 22593 sys_catalog.cc:564] T 00000000000000000000000000000000 P c40e7cffb88d4c38b95f967c984fe80f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:05:02.459949 22604 sys_catalog.cc:455] T 00000000000000000000000000000000 P c40e7cffb88d4c38b95f967c984fe80f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c40e7cffb88d4c38b95f967c984fe80f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c40e7cffb88d4c38b95f967c984fe80f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36805 } } }
I20260929 21:05:02.460268 22604 sys_catalog.cc:458] T 00000000000000000000000000000000 P c40e7cffb88d4c38b95f967c984fe80f [sys.catalog]: This master's current role is: LEADER
I20260929 21:05:02.461117 22609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:05:02.460973 22605 sys_catalog.cc:455] T 00000000000000000000000000000000 P c40e7cffb88d4c38b95f967c984fe80f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c40e7cffb88d4c38b95f967c984fe80f. Latest consensus state: current_term: 1 leader_uuid: "c40e7cffb88d4c38b95f967c984fe80f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c40e7cffb88d4c38b95f967c984fe80f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36805 } } }
I20260929 21:05:02.461336 22605 sys_catalog.cc:458] T 00000000000000000000000000000000 P c40e7cffb88d4c38b95f967c984fe80f [sys.catalog]: This master's current role is: LEADER
I20260929 21:05:02.467456 22609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:05:02.483412 22609 catalog_manager.cc:1347] Generated new cluster ID: 61ad218071864534adce82e6eb86278f
I20260929 21:05:02.483516 22609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:05:02.512225 22609 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:05:02.513619 22609 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:05:02.521514 22609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c40e7cffb88d4c38b95f967c984fe80f: Generated new TSK 0
I20260929 21:05:02.522555 22609 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 21:05:02.531497 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 22526
W20260929 21:05:02.532557 22588 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:32997
W20260929 21:05:02.533280 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 22526: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 21:05:02.533427 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 22526
2026-09-29T21:05:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1886 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 21:05:02 dist-test-slave-1dms krb5kdc[22624](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 21:05:02 dist-test-slave-1dms krb5kdc[22624](info): set up 2 sockets
Sep 29 21:05:02 dist-test-slave-1dms krb5kdc[22624](info): commencing operation
krb5kdc: starting...
W20260929 21:05:03.629455 388 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.009s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 21:05:03 dist-test-slave-1dms krb5kdc[22624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T21:05:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:05:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 21:05:03.876082 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42955
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:05:04.201511 22637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:05:04.201800 22637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:05:04.201998 22637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:05:04.209790 22637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:05:04.209926 22637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:05:04.210021 22637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:05:04.210119 22637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:05:04.210191 22637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:05:04.219956 22637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:05:04.221292 22637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:05:04.222564 22637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:05:04.233983 22642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:05:04.233948 22643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:05:04.234005 22645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:05:04.234637 22637 server_base.cc:1034] running on GCE node
I20260929 21:05:04.235469 22637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:05:04.236812 22637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:05:04.238086 22637 hybrid_clock.cc:648] HybridClock initialized: now 1790715904238034 us; error 51 us; skew 500 ppm
I20260929 21:05:04.238422 22637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:05:04 dist-test-slave-1dms krb5kdc[22624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:05:04.248010 22637 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 21:05:04.249696 22637 webserver.cc:460] Webserver started at http://127.0.97.62:33895/ using document root <none> and password file <none>
I20260929 21:05:04.250425 22637 fs_manager.cc:359] Metadata directory not provided
I20260929 21:05:04.250573 22637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:05:04.250890 22637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:05:04.253926 22637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "a967d3028cd9421f9d87d48c49441659"
format_stamp: "Formatted at 2026-09-29 21:05:04 on dist-test-slave-1dms"
server_key: "b12d55975b82af974e666dd6816c876c"
server_key_iv: "10c1b6fd6d675859dda9ce3f8a5f23da"
server_key_version: "encryptionkey@0"
I20260929 21:05:04.254688 22637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "a967d3028cd9421f9d87d48c49441659"
format_stamp: "Formatted at 2026-09-29 21:05:04 on dist-test-slave-1dms"
server_key: "b12d55975b82af974e666dd6816c876c"
server_key_iv: "10c1b6fd6d675859dda9ce3f8a5f23da"
server_key_version: "encryptionkey@0"
I20260929 21:05:04.259732 22637 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 21:05:04.262941 22652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:05:04.264118 22637 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 21:05:04.264286 22637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "a967d3028cd9421f9d87d48c49441659"
format_stamp: "Formatted at 2026-09-29 21:05:04 on dist-test-slave-1dms"
server_key: "b12d55975b82af974e666dd6816c876c"
server_key_iv: "10c1b6fd6d675859dda9ce3f8a5f23da"
server_key_version: "encryptionkey@0"
I20260929 21:05:04.264472 22637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:05:04.270061 22637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:05:04.273751 22637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:05:04.274034 22637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:05:04.292309 22637 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42955
I20260929 21:05:04.292357 22703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42955 every 8 connection(s)
I20260929 21:05:04.293725 22637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:05:04.297263 22704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:05:04.299903 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 22637
I20260929 21:05:04.300071 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:05:04.300604 388 external_mini_cluster.cc:1428] Setting key 9b077fbd71a885bd644c47fcab46ad46
I20260929 21:05:04.308404 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:04.302491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45697 (local address 127.0.97.62:42955)
0929 21:05:04.303327 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:04.303339 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:04.303386 (+ 47us) server_negotiation.cc:407] Connection header received
0929 21:05:04.305047 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:04.305070 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:04.305406 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:04.305796 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:04.306830 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:05:04.306836 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:05:04.306848 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:05:04.306860 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:05:04.306914 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:05:04.306935 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:05:04.307186 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:04.307330 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 21:05:04.307506 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":390,"threads_started":1}
I20260929 21:05:04.309357 22704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a967d3028cd9421f9d87d48c49441659: Bootstrap starting.
I20260929 21:05:04.311971 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 21:05:04.312613 22704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a967d3028cd9421f9d87d48c49441659: Neither blocks nor log segments found. Creating new log.
I20260929 21:05:04.313670 22704 log.cc:824] T 00000000000000000000000000000000 P a967d3028cd9421f9d87d48c49441659: Log is configured to *not* fsync() on all Append() calls
I20260929 21:05:04.316713 22704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a967d3028cd9421f9d87d48c49441659: No bootstrap required, opened a new log
I20260929 21:05:04.319150 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:04.316397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57049 (local address 127.0.97.62:42955)
0929 21:05:04.316567 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:04.316573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:04.317173 (+ 600us) server_negotiation.cc:407] Connection header received
0929 21:05:04.317375 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:04.317379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:04.317423 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:04.317498 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:04.318489 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:05:04.318493 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:05:04.318496 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 21:05:04.318511 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:05:04.318552 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 21:05:04.318567 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:05:04.318664 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:04.318788 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 21:05:04.318837 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 21:05:04.325314 22704 raft_consensus.cc:348] T 00000000000000000000000000000000 P a967d3028cd9421f9d87d48c49441659 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a967d3028cd9421f9d87d48c49441659" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42955 } }
I20260929 21:05:04.325695 22704 raft_consensus.cc:374] T 00000000000000000000000000000000 P a967d3028cd9421f9d87d48c49441659 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 21:05:04.325810 22704 raft_consensus.cc:729] T 00000000000000000000000000000000 P a967d3028cd9421f9d87d48c49441659 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a967d3028cd9421f9d87d48c49441659, State: Initialized, Role: FOLLOWER
I20260929 21:05:04.326462 22704 consensus_queue.cc:260] T 00000000000000000000000000000000 P a967d3028cd9421f9d87d48c49441659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a967d3028cd9421f9d87d48c49441659" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42955 } }
I20260929 21:05:04.326748 22704 raft_consensus.cc:388] T 00000000000000000000000000000000 P a967d3028cd9421f9d87d48c49441659 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 21:05:04.326912 22704 raft_consensus.cc:482] T 00000000000000000000000000000000 P a967d3028cd9421f9d87d48c49441659 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 21:05:04.327078 22704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a967d3028cd9421f9d87d48c49441659 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:05:04.329257 22704 raft_consensus.cc:504] T 00000000000000000000000000000000 P a967d3028cd9421f9d87d48c49441659 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a967d3028cd9421f9d87d48c49441659" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42955 } }
I20260929 21:05:04.329731 22704 leader_election.cc:302] T 00000000000000000000000000000000 P a967d3028cd9421f9d87d48c49441659 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a967d3028cd9421f9d87d48c49441659; no voters:
I20260929 21:05:04.330374 22704 leader_election.cc:288] T 00000000000000000000000000000000 P a967d3028cd9421f9d87d48c49441659 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 21:05:04.330489 22714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a967d3028cd9421f9d87d48c49441659 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:05:04.331866 22714 raft_consensus.cc:686] T 00000000000000000000000000000000 P a967d3028cd9421f9d87d48c49441659 [term 1 LEADER]: Becoming Leader. State: Replica: a967d3028cd9421f9d87d48c49441659, State: Running, Role: LEADER
I20260929 21:05:04.332480 22714 consensus_queue.cc:237] T 00000000000000000000000000000000 P a967d3028cd9421f9d87d48c49441659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a967d3028cd9421f9d87d48c49441659" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42955 } }
I20260929 21:05:04.332672 22704 sys_catalog.cc:564] T 00000000000000000000000000000000 P a967d3028cd9421f9d87d48c49441659 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:05:04.335314 22716 sys_catalog.cc:455] T 00000000000000000000000000000000 P a967d3028cd9421f9d87d48c49441659 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a967d3028cd9421f9d87d48c49441659. Latest consensus state: current_term: 1 leader_uuid: "a967d3028cd9421f9d87d48c49441659" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a967d3028cd9421f9d87d48c49441659" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42955 } } }
I20260929 21:05:04.335419 22715 sys_catalog.cc:455] T 00000000000000000000000000000000 P a967d3028cd9421f9d87d48c49441659 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a967d3028cd9421f9d87d48c49441659" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a967d3028cd9421f9d87d48c49441659" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42955 } } }
I20260929 21:05:04.335940 22716 sys_catalog.cc:458] T 00000000000000000000000000000000 P a967d3028cd9421f9d87d48c49441659 [sys.catalog]: This master's current role is: LEADER
I20260929 21:05:04.335980 22715 sys_catalog.cc:458] T 00000000000000000000000000000000 P a967d3028cd9421f9d87d48c49441659 [sys.catalog]: This master's current role is: LEADER
I20260929 21:05:04.339252 22723 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:05:04.344202 22723 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:05:04.353236 22723 catalog_manager.cc:1347] Generated new cluster ID: 9ee7d9fbe7984019a041065a7cf7995a
I20260929 21:05:04.353370 22723 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:05:04.365984 22723 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:05:04.366451 22723 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:05:04.373914 22723 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a967d3028cd9421f9d87d48c49441659: Generated new TSK 0
I20260929 21:05:04.374680 22723 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 21:05:04.384248 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 22637
W20260929 21:05:04.385504 22699 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45697
W20260929 21:05:04.386142 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 22637: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 21:05:04.386293 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 22637
2026-09-29T21:05:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1852 ms)
[----------] 18 tests from AuthTokenIssuingTest (34570 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 21:05:04 dist-test-slave-1dms krb5kdc[22735](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 21:05:04 dist-test-slave-1dms krb5kdc[22735](info): set up 2 sockets
Sep 29 21:05:04 dist-test-slave-1dms krb5kdc[22735](info): commencing operation
krb5kdc: starting...
W20260929 21:05:05.501497 388 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 21:05:05 dist-test-slave-1dms krb5kdc[22735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T21:05:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:05:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 21:05:05.750380 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38069
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38069,127.0.97.61:34517,127.0.97.60:42019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:05:06.071244 22748 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 21:05:06.071568 22748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:05:06.071645 22748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:05:06.071725 22748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:05:06.078729 22748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:05:06.078814 22748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:05:06.078869 22748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:05:06.078920 22748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:05:06.078981 22748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:05:06.088397 22748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38069,127.0.97.61:34517,127.0.97.60:42019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:05:06.089568 22748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:05:06.090767 22748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:05:06.101982 22754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:05:06.102077 22753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:05:06.102051 22756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:05:06.103060 22748 server_base.cc:1034] running on GCE node
I20260929 21:05:06.103729 22748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:05:06.105027 22748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:05:06.106238 22748 hybrid_clock.cc:648] HybridClock initialized: now 1790715906106189 us; error 24 us; skew 500 ppm
I20260929 21:05:06.106519 22748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:05:06 dist-test-slave-1dms krb5kdc[22735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715906, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:05:06.109495 22748 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 21:05:06.110821 22748 webserver.cc:460] Webserver started at http://127.0.97.62:33489/ using document root <none> and password file <none>
I20260929 21:05:06.111496 22748 fs_manager.cc:359] Metadata directory not provided
I20260929 21:05:06.111609 22748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:05:06.111955 22748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:05:06.114871 22748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "a53e0908b41f4ff49f73f3a53c2dc0c2"
format_stamp: "Formatted at 2026-09-29 21:05:06 on dist-test-slave-1dms"
server_key: "2180118887e7def291b9d8f0638f38c5"
server_key_iv: "8f1e8f6ecd5a3219357e2dd6d6c559c1"
server_key_version: "encryptionkey@0"
I20260929 21:05:06.115577 22748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "a53e0908b41f4ff49f73f3a53c2dc0c2"
format_stamp: "Formatted at 2026-09-29 21:05:06 on dist-test-slave-1dms"
server_key: "2180118887e7def291b9d8f0638f38c5"
server_key_iv: "8f1e8f6ecd5a3219357e2dd6d6c559c1"
server_key_version: "encryptionkey@0"
I20260929 21:05:06.120486 22748 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260929 21:05:06.123921 22763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:05:06.125268 22748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260929 21:05:06.125459 22748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "a53e0908b41f4ff49f73f3a53c2dc0c2"
format_stamp: "Formatted at 2026-09-29 21:05:06 on dist-test-slave-1dms"
server_key: "2180118887e7def291b9d8f0638f38c5"
server_key_iv: "8f1e8f6ecd5a3219357e2dd6d6c559c1"
server_key_version: "encryptionkey@0"
I20260929 21:05:06.125676 22748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:05:06.143412 22748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:05:06.147776 22748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:05:06.148061 22748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:05:06.166533 22748 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38069
I20260929 21:05:06.166566 22814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38069 every 8 connection(s)
I20260929 21:05:06.167959 22748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:05:06.171494 22815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:05:06.173318 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 22748
I20260929 21:05:06.173538 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:05:06.174167 388 external_mini_cluster.cc:1428] Setting key 0baa3ba2adcdf4d8bb93f2da49a512ef
I20260929 21:05:06.178054 22815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38069 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 21:05:06 dist-test-slave-1dms krb5kdc[22735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715906, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260929 21:05:06.203215 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:06.181621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38069 (local address 127.0.0.1:60446)
0929 21:05:06.182459 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:06.182505 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 21:05:06.185284 (+ 2779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:05:06.186128 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:05:06.186137 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:05:06.187009 (+ 872us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:05:06.187516 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:05:06.187542 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:05:06.189167 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:05:06.189175 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:05:06.191041 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:05:06.191058 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:05:06.191956 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:05:06.191965 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:05:06.192184 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:06.192908 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:05:06.192941 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:05:06.195709 (+ 2768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:05:06.198856 (+ 3147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:05:06.198868 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:05:06.198880 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:05:06.199288 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:05:06.199666 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:05:06.199673 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:05:06.199676 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:05:06.199827 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:05:06.200602 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:05:06.200609 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:05:06.200815 (+ 206us) client_negotiation.cc:769] Sending connection context
0929 21:05:06.201215 (+ 400us) client_negotiation.cc:240] Negotiation successful
0929 21:05:06.201873 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":200,"thread_start_us":415,"threads_started":1}
I20260929 21:05:06.203244 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:06.181215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60446 (local address 127.0.97.62:38069)
0929 21:05:06.182044 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:06.182059 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:06.182648 (+ 589us) server_negotiation.cc:407] Connection header received
0929 21:05:06.185642 (+ 2994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:06.185648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:06.185872 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:06.186111 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:06.187731 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:06.189013 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:06.191179 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:06.191711 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:06.191841 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:06.195942 (+ 4101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:05:06.195949 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:05:06.195960 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:05:06.195992 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:05:06.198726 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:06.199400 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:06.199409 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:06.199415 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:06.199558 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:06.199935 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:06.199941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:06.199944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:06.200464 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:05:06.200669 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:06.201283 (+ 614us) server_negotiation.cc:299] Negotiation successful
0929 21:05:06.201761 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":596,"thread_start_us":545,"threads_started":1}
I20260929 21:05:06.206190 22815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34517 } has no permanent_uuid. Determining permanent_uuid...
I20260929 21:05:06.207160 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:06.206786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34517 (local address 127.0.0.1:45736)
0929 21:05:06.206943 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:06.207056 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 21:05:06.207384 22764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34517: connect: Connection refused (error 111)
I20260929 21:05:06.208957 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:06.208547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34517 (local address 127.0.0.1:45750)
0929 21:05:06.208695 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:06.208766 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 21:05:06.209154 22815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34517: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34517: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260929 21:05:06.231969 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:34517
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38069,127.0.97.61:34517,127.0.97.60:42019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 21:05:06.258616 22815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34517 } attempt: 1
I20260929 21:05:06.259769 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:06.259318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34517 (local address 127.0.0.1:45766)
0929 21:05:06.259549 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:06.259646 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260929 21:05:06.260777 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:06.260446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34517 (local address 127.0.0.1:45774)
0929 21:05:06.260599 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:06.260659 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 21:05:06.261024 22815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34517: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34517: connect: Connection refused (error 111)
I20260929 21:05:06.329378 22815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34517 } attempt: 2
I20260929 21:05:06.330595 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:06.330053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34517 (local address 127.0.0.1:45782)
0929 21:05:06.330315 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:06.330438 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 21:05:06.331630 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:06.331292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34517 (local address 127.0.0.1:45794)
0929 21:05:06.331432 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:06.331514 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 21:05:06.331887 22815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34517: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34517: connect: Connection refused (error 111)
I20260929 21:05:06.423167 22815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34517 } attempt: 3
I20260929 21:05:06.424360 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:06.423914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34517 (local address 127.0.0.1:45804)
0929 21:05:06.424123 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:06.424222 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 21:05:06.425594 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:06.425241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34517 (local address 127.0.0.1:45816)
0929 21:05:06.425376 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:06.425459 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 21:05:06.425834 22815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34517: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34517: connect: Connection refused (error 111)
W20260929 21:05:06.559257 22822 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 21:05:06.559634 22822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:05:06.559759 22822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:05:06.559912 22822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:05:06.567250 22822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:05:06.567341 22822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:05:06.567466 22822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:05:06.567544 22822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:05:06.567659 22822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:05:06.569082 22815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34517 } attempt: 4
I20260929 21:05:06.570351 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:06.569824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34517 (local address 127.0.0.1:45832)
0929 21:05:06.570135 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:06.570223 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260929 21:05:06.571338 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:06.571032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34517 (local address 127.0.0.1:45838)
0929 21:05:06.571154 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:06.571229 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 21:05:06.571599 22815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34517: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34517: connect: Connection refused (error 111)
I20260929 21:05:06.578379 22822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38069,127.0.97.61:34517,127.0.97.60:42019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:34517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:05:06.579654 22822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:05:06.581035 22822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:05:06.593202 22827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:05:06.593173 22830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:05:06.593127 22828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:05:06.594050 22822 server_base.cc:1034] running on GCE node
I20260929 21:05:06.594781 22822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:05:06.596134 22822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:05:06.597352 22822 hybrid_clock.cc:648] HybridClock initialized: now 1790715906597280 us; error 38 us; skew 500 ppm
I20260929 21:05:06.597697 22822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:05:06 dist-test-slave-1dms krb5kdc[22735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715906, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:05:06.600822 22822 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260929 21:05:06.602289 22822 webserver.cc:460] Webserver started at http://127.0.97.61:40563/ using document root <none> and password file <none>
I20260929 21:05:06.602958 22822 fs_manager.cc:359] Metadata directory not provided
I20260929 21:05:06.603070 22822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:05:06.603421 22822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:05:06.606294 22822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-1/data/instance:
uuid: "3ebc991527464415a4739345993e44e9"
format_stamp: "Formatted at 2026-09-29 21:05:06 on dist-test-slave-1dms"
server_key: "4191a77b4fc52db9a1e1fe781e701d39"
server_key_iv: "2c792a5d7414f33d6d41c4239bf9db18"
server_key_version: "encryptionkey@0"
I20260929 21:05:06.607074 22822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-1/wal/instance:
uuid: "3ebc991527464415a4739345993e44e9"
format_stamp: "Formatted at 2026-09-29 21:05:06 on dist-test-slave-1dms"
server_key: "4191a77b4fc52db9a1e1fe781e701d39"
server_key_iv: "2c792a5d7414f33d6d41c4239bf9db18"
server_key_version: "encryptionkey@0"
I20260929 21:05:06.611883 22822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260929 21:05:06.615140 22837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:05:06.616410 22822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 21:05:06.616607 22822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-1/wal
uuid: "3ebc991527464415a4739345993e44e9"
format_stamp: "Formatted at 2026-09-29 21:05:06 on dist-test-slave-1dms"
server_key: "4191a77b4fc52db9a1e1fe781e701d39"
server_key_iv: "2c792a5d7414f33d6d41c4239bf9db18"
server_key_version: "encryptionkey@0"
I20260929 21:05:06.616824 22822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:05:06.646935 22822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:05:06.651374 22822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:05:06.651645 22822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:05:06.669579 22822 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:34517
I20260929 21:05:06.669603 22888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:34517 every 8 connection(s)
I20260929 21:05:06.670878 22822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-1/data/info.pb
I20260929 21:05:06.674431 22889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:05:06.675115 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 22822
I20260929 21:05:06.675297 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-1/wal/instance
I20260929 21:05:06.675876 388 external_mini_cluster.cc:1428] Setting key 6bbb8d5165ef07938bcbd452345a3713
I20260929 21:05:06.684334 22889 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38069 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 21:05:06 dist-test-slave-1dms krb5kdc[22735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715906, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260929 21:05:06.708721 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:06.687577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60448 (local address 127.0.97.62:38069)
0929 21:05:06.687796 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:06.687802 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:06.689093 (+ 1291us) server_negotiation.cc:407] Connection header received
0929 21:05:06.691929 (+ 2836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:06.691933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:06.692001 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:06.692089 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:06.694173 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:06.694829 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:06.697280 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:06.697721 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:06.697816 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:06.701931 (+ 4115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:05:06.701939 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:05:06.701945 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:05:06.701985 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:05:06.705334 (+ 3349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:06.706081 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:06.706085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:06.706087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:06.706208 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:06.706682 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:06.706685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:06.706687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:06.707023 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:05:06.707173 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:06.708010 (+ 837us) server_negotiation.cc:299] Negotiation successful
0929 21:05:06.708271 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 21:05:06.709352 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:06.688060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38069 (local address 127.0.0.1:60448)
0929 21:05:06.688920 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:06.688974 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 21:05:06.691635 (+ 2661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:05:06.692358 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:05:06.692372 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:05:06.693177 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:05:06.693993 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:05:06.694019 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:05:06.695003 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:05:06.695011 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:05:06.697117 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:05:06.697139 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:05:06.697942 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:05:06.697951 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:05:06.698262 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:06.699001 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:05:06.699035 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:05:06.701692 (+ 2657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:05:06.705454 (+ 3762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:05:06.705461 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:05:06.705473 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:05:06.705940 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:05:06.706326 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:05:06.706334 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:05:06.706340 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:05:06.706564 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:05:06.707193 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:05:06.707204 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:05:06.707620 (+ 416us) client_negotiation.cc:769] Sending connection context
0929 21:05:06.707842 (+ 222us) client_negotiation.cc:240] Negotiation successful
0929 21:05:06.708215 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":312,"threads_started":1}
I20260929 21:05:06.711166 22889 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34517 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 21:05:06 dist-test-slave-1dms krb5kdc[22735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715906, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 21:05:06.725481 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:06.711815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34517 (local address 127.0.0.1:45840)
0929 21:05:06.711968 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:06.711990 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 21:05:06.712079 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:05:06.713175 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:05:06.713179 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:05:06.713614 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:05:06.713747 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:05:06.713755 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:05:06.715097 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:05:06.715103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:05:06.715930 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:05:06.715941 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:05:06.716718 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:05:06.716723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:05:06.716936 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:06.717519 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:05:06.717535 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:05:06.719445 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:05:06.723012 (+ 3567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:05:06.723020 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:05:06.723025 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:05:06.723374 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:05:06.723744 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:05:06.723748 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:05:06.723752 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:05:06.723916 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:05:06.724480 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:05:06.724489 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:05:06.724616 (+ 127us) client_negotiation.cc:769] Sending connection context
0929 21:05:06.724776 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 21:05:06.725062 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 21:05:06.725538 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:06.712105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45840 (local address 127.0.97.61:34517)
0929 21:05:06.712719 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:06.712725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:06.712748 (+ 23us) server_negotiation.cc:407] Connection header received
0929 21:05:06.712786 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:06.712792 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:06.713033 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:06.713192 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:06.713927 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:06.714946 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:06.716063 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:06.716580 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:06.716713 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:06.719655 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:05:06.719666 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:05:06.719678 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:05:06.719704 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:05:06.722884 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:06.723508 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:06.723517 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:06.723522 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:06.723633 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:06.724026 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:06.724031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:06.724034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:06.724356 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:05:06.724565 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:06.724932 (+ 367us) server_negotiation.cc:299] Negotiation successful
0929 21:05:06.725170 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":470,"thread_start_us":347,"threads_started":1}
I20260929 21:05:06.728308 22889 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42019 } has no permanent_uuid. Determining permanent_uuid...
I20260929 21:05:06.729542 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:06.729051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42019 (local address 127.0.0.1:55774)
0929 21:05:06.729207 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:06.729410 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 21:05:06.729833 22841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42019: connect: Connection refused (error 111)
I20260929 21:05:06.731298 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:06.730955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42019 (local address 127.0.0.1:55778)
0929 21:05:06.731131 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:06.731202 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 21:05:06.731508 22889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42019: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42019: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260929 21:05:06.738351 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:42019
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38069,127.0.97.61:34517,127.0.97.60:42019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 21:05:06.780900 22889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42019 } attempt: 1
I20260929 21:05:06.782174 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:06.781610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42019 (local address 127.0.0.1:55788)
0929 21:05:06.781843 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:06.781967 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 21:05:06.783325 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:06.783007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42019 (local address 127.0.0.1:55796)
0929 21:05:06.783161 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:06.783229 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 21:05:06.783624 22889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42019: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42019: connect: Connection refused (error 111)
I20260929 21:05:06.851938 22889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42019 } attempt: 2
I20260929 21:05:06.853173 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:06.852673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42019 (local address 127.0.0.1:55812)
0929 21:05:06.852927 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:06.853018 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 21:05:06.854130 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:06.853857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42019 (local address 127.0.0.1:55818)
0929 21:05:06.853975 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:06.854033 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 21:05:06.854343 22889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42019: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42019: connect: Connection refused (error 111)
I20260929 21:05:06.870862 22815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34517 } attempt: 5
Sep 29 21:05:06 dist-test-slave-1dms krb5kdc[22735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715906, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 21:05:06.885210 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:06.871553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34517 (local address 127.0.0.1:45852)
0929 21:05:06.871745 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:06.871772 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 21:05:06.871874 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:05:06.872251 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:05:06.872254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:05:06.872628 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:05:06.872798 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:05:06.872807 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:05:06.874282 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:05:06.874287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:05:06.875148 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:05:06.875160 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:05:06.876163 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:05:06.876167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:05:06.876310 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:06.876968 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:05:06.876987 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:05:06.879151 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:05:06.883032 (+ 3881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:05:06.883036 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:05:06.883039 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:05:06.883398 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:05:06.883772 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:05:06.883775 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:05:06.883778 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:05:06.883899 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:05:06.884468 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:05:06.884472 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:05:06.884594 (+ 122us) client_negotiation.cc:769] Sending connection context
0929 21:05:06.884697 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 21:05:06.884949 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 21:05:06.885399 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:06.871662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45852 (local address 127.0.97.61:34517)
0929 21:05:06.871858 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:06.871866 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:06.871909 (+ 43us) server_negotiation.cc:407] Connection header received
0929 21:05:06.871987 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:06.871994 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:06.872087 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:06.872227 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:06.873004 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:06.874142 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:06.875323 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:06.876027 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:06.876156 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:06.879364 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:05:06.879373 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:05:06.879378 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:05:06.879426 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:05:06.882880 (+ 3454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:06.883515 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:06.883521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:06.883524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:06.883666 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:06.884000 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:06.884005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:06.884008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:06.884354 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:05:06.884498 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:06.884859 (+ 361us) server_negotiation.cc:299] Negotiation successful
0929 21:05:06.885094 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 21:05:06.886365 22815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42019 } has no permanent_uuid. Determining permanent_uuid...
I20260929 21:05:06.887530 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:06.887130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42019 (local address 127.0.0.1:55828)
0929 21:05:06.887299 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:06.887391 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 21:05:06.888491 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:06.888167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42019 (local address 127.0.0.1:55832)
0929 21:05:06.888303 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:06.888400 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 21:05:06.888715 22815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42019: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42019: connect: Connection refused (error 111)
I20260929 21:05:06.939985 22815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42019 } attempt: 1
I20260929 21:05:06.941151 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:06.940652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42019 (local address 127.0.0.1:55848)
0929 21:05:06.940876 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:06.940979 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 21:05:06.942220 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:06.941810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42019 (local address 127.0.0.1:55858)
0929 21:05:06.941991 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:06.942086 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 21:05:06.942443 22815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42019: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42019: connect: Connection refused (error 111)
I20260929 21:05:06.945538 22889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42019 } attempt: 3
I20260929 21:05:06.946547 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:06.946147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42019 (local address 127.0.0.1:55874)
0929 21:05:06.946330 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:06.946410 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 21:05:06.947540 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:06.947155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42019 (local address 127.0.0.1:55880)
0929 21:05:06.947363 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:06.947442 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260929 21:05:06.947738 22889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42019: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42019: connect: Connection refused (error 111)
I20260929 21:05:07.010684 22815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42019 } attempt: 2
I20260929 21:05:07.011967 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:07.011375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42019 (local address 127.0.0.1:55894)
0929 21:05:07.011633 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:07.011762 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 21:05:07.013044 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:07.012612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42019 (local address 127.0.0.1:55900)
0929 21:05:07.012773 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:07.012885 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 21:05:07.013327 22815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42019: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42019: connect: Connection refused (error 111)
W20260929 21:05:07.069350 22896 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 21:05:07.069636 22896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:05:07.069706 22896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:05:07.069794 22896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:05:07.076720 22896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:05:07.076810 22896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:05:07.076923 22896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:05:07.076980 22896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:05:07.077041 22896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:05:07.086623 22896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38069,127.0.97.61:34517,127.0.97.60:42019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:42019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:05:07.087831 22896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:05:07.089094 22896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 21:05:07.090924 22889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42019 } attempt: 4
I20260929 21:05:07.091970 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:07.091513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42019 (local address 127.0.0.1:55916)
0929 21:05:07.091733 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:07.091813 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 21:05:07.092952 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:07.092596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42019 (local address 127.0.0.1:55922)
0929 21:05:07.092733 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:07.092798 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 21:05:07.093160 22889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42019: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42019: connect: Connection refused (error 111)
W20260929 21:05:07.100492 22901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:05:07.100538 22902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:05:07.100597 22904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:05:07.101643 22896 server_base.cc:1034] running on GCE node
I20260929 21:05:07.102432 22896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:05:07.103776 22896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:05:07.105013 22896 hybrid_clock.cc:648] HybridClock initialized: now 1790715907104941 us; error 48 us; skew 500 ppm
I20260929 21:05:07.105335 22896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:05:07 dist-test-slave-1dms krb5kdc[22735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715907, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:05:07.108331 22896 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260929 21:05:07.109766 22896 webserver.cc:460] Webserver started at http://127.0.97.60:34343/ using document root <none> and password file <none>
I20260929 21:05:07.110459 22896 fs_manager.cc:359] Metadata directory not provided
I20260929 21:05:07.110589 22896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:05:07.110910 22896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:05:07.113845 22896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-2/data/instance:
uuid: "f422674aaaa349c1afd192f1eb5b564e"
format_stamp: "Formatted at 2026-09-29 21:05:07 on dist-test-slave-1dms"
server_key: "3304299f2e64c5eb932d7e038793dac4"
server_key_iv: "cc1bd0b505f6d61edaf1b2a90cf2799f"
server_key_version: "encryptionkey@0"
I20260929 21:05:07.114611 22896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-2/wal/instance:
uuid: "f422674aaaa349c1afd192f1eb5b564e"
format_stamp: "Formatted at 2026-09-29 21:05:07 on dist-test-slave-1dms"
server_key: "3304299f2e64c5eb932d7e038793dac4"
server_key_iv: "cc1bd0b505f6d61edaf1b2a90cf2799f"
server_key_version: "encryptionkey@0"
I20260929 21:05:07.119423 22896 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260929 21:05:07.119580 22815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42019 } attempt: 3
I20260929 21:05:07.120735 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:07.120304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42019 (local address 127.0.0.1:55924)
0929 21:05:07.120505 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:07.120586 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 21:05:07.121865 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:07.121490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42019 (local address 127.0.0.1:55932)
0929 21:05:07.121701 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:07.121764 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260929 21:05:07.122092 22815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42019: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42019: connect: Connection refused (error 111)
I20260929 21:05:07.123304 22911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:05:07.124607 22896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 21:05:07.124785 22896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-2/wal
uuid: "f422674aaaa349c1afd192f1eb5b564e"
format_stamp: "Formatted at 2026-09-29 21:05:07 on dist-test-slave-1dms"
server_key: "3304299f2e64c5eb932d7e038793dac4"
server_key_iv: "cc1bd0b505f6d61edaf1b2a90cf2799f"
server_key_version: "encryptionkey@0"
I20260929 21:05:07.125046 22896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:05:07.149431 22896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:05:07.153268 22896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:05:07.153541 22896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:05:07.171453 22896 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:42019
I20260929 21:05:07.171501 22962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:42019 every 8 connection(s)
I20260929 21:05:07.172734 22896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-2/data/info.pb
I20260929 21:05:07.176708 22963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:05:07.182098 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 22896
I20260929 21:05:07.182297 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790715556572573-388-0/minicluster-data/master-2/wal/instance
I20260929 21:05:07.182689 388 external_mini_cluster.cc:1428] Setting key 192e03b5044eefc1b9075429adb9f0ee
I20260929 21:05:07.182801 22963 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38069 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 21:05:07 dist-test-slave-1dms krb5kdc[22735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715907, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 29 21:05:07 dist-test-slave-1dms krb5kdc[22735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 21:05:07.202328 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:07.185665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60458 (local address 127.0.97.62:38069)
0929 21:05:07.185874 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:07.185880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:07.186569 (+ 689us) server_negotiation.cc:407] Connection header received
0929 21:05:07.187496 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:07.187504 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:07.187578 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:07.187698 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:07.189326 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:07.190019 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:07.191592 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:07.192005 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:07.192130 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:07.195626 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:05:07.195631 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:05:07.195635 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:05:07.195665 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:05:07.199021 (+ 3356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:07.199874 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:07.199890 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:07.199894 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:07.200047 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:07.200394 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:07.200397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:07.200399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:07.200648 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:05:07.200758 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:07.201699 (+ 941us) server_negotiation.cc:299] Negotiation successful
0929 21:05:07.201937 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 21:05:07.203267 22967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:07.185573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38069 (local address 127.0.0.1:60458)
0929 21:05:07.186419 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:07.186470 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 21:05:07.187263 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:05:07.187915 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:05:07.187922 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:05:07.188595 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:05:07.189151 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:05:07.189177 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:05:07.190165 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:05:07.190171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:05:07.191455 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:05:07.191466 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:05:07.192247 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:05:07.192255 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:05:07.192430 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:07.193203 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:05:07.193233 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:05:07.195402 (+ 2169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:05:07.199219 (+ 3817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:05:07.199229 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:05:07.199236 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:05:07.199752 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:05:07.200140 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:05:07.200143 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:05:07.200145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:05:07.200299 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:05:07.200823 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:05:07.200832 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:05:07.201271 (+ 439us) client_negotiation.cc:769] Sending connection context
0929 21:05:07.201590 (+ 319us) client_negotiation.cc:240] Negotiation successful
0929 21:05:07.201941 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"spinlock_wait_cycles":11008,"thread_start_us":431,"threads_started":1}
I20260929 21:05:07.203780 22966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:07.184609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55940 (local address 127.0.97.60:42019)
0929 21:05:07.185343 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:07.185358 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:07.185406 (+ 48us) server_negotiation.cc:407] Connection header received
0929 21:05:07.187254 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:07.187273 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:07.187537 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:07.187746 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:07.189189 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:07.190524 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:07.191743 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:07.192263 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:07.192436 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:07.196530 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:05:07.196540 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:05:07.196555 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:05:07.196589 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:05:07.200036 (+ 3447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:07.201073 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:07.201078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:07.201081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:07.201197 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:07.201603 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:07.201607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:07.201611 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:07.201990 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:05:07.202297 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:07.203151 (+ 854us) server_negotiation.cc:299] Negotiation successful
0929 21:05:07.203415 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":544,"thread_start_us":352,"threads_started":1}
I20260929 21:05:07.204910 22963 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34517 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 21:05:07 dist-test-slave-1dms krb5kdc[22735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715907, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 29 21:05:07 dist-test-slave-1dms krb5kdc[22735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:05:07.217965 22967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:07.205457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34517 (local address 127.0.0.1:45854)
0929 21:05:07.205912 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:07.205930 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 21:05:07.206036 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:05:07.206427 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:05:07.206435 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:05:07.206858 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:05:07.206991 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:05:07.206999 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:05:07.207939 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:05:07.207947 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:05:07.208806 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:05:07.208817 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:05:07.209550 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:05:07.209555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:05:07.209736 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:07.210271 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:05:07.210287 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:05:07.212348 (+ 2061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:05:07.215565 (+ 3217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:05:07.215572 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:05:07.215575 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:05:07.215928 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:05:07.216319 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:05:07.216324 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:05:07.216328 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:05:07.216485 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:05:07.217048 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:05:07.217053 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:05:07.217197 (+ 144us) client_negotiation.cc:769] Sending connection context
0929 21:05:07.217338 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 21:05:07.217597 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339}
I20260929 21:05:07.217998 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:07.205733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45854 (local address 127.0.97.61:34517)
0929 21:05:07.205932 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:07.205938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:07.205980 (+ 42us) server_negotiation.cc:407] Connection header received
0929 21:05:07.206158 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:07.206164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:07.206240 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:07.206339 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:07.207132 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:07.207795 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:07.208990 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:07.209407 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:07.209544 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:07.212535 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:05:07.212540 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:05:07.212543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:05:07.212574 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:05:07.215436 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:07.216052 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:07.216063 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:07.216067 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:07.216207 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:07.216593 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:07.216600 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:07.216603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:07.216934 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:05:07.217085 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:07.217446 (+ 361us) server_negotiation.cc:299] Negotiation successful
0929 21:05:07.217648 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 21:05:07.218948 22963 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42019 } has no permanent_uuid. Determining permanent_uuid...
I20260929 21:05:07.219887 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:07.206718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60460 (local address 127.0.97.62:38069)
0929 21:05:07.206891 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:07.206896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:07.206912 (+ 16us) server_negotiation.cc:407] Connection header received
0929 21:05:07.207004 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:07.207009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:07.207072 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:07.207168 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:07.208215 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:07.209009 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:07.210179 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:07.210589 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:07.210718 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:07.214508 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:05:07.214521 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:05:07.214525 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:05:07.214553 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:05:07.217122 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:07.217722 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:07.217729 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:07.217732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:07.217857 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:07.218271 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:07.218276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:07.218278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:07.218651 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:05:07.218921 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:07.219302 (+ 381us) server_negotiation.cc:299] Negotiation successful
0929 21:05:07.219548 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 21:05:07 dist-test-slave-1dms krb5kdc[22735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715907, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 29 21:05:07 dist-test-slave-1dms krb5kdc[22735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 21:05:07.233493 22967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:07.219762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42019 (local address 127.0.0.1:55942)
0929 21:05:07.219943 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:07.219968 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 21:05:07.220103 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:05:07.220735 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:05:07.220738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:05:07.221210 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:05:07.221353 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:05:07.221362 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:05:07.222620 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:05:07.222624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:05:07.223559 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:05:07.223572 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:05:07.224248 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:05:07.224255 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:05:07.224462 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:07.225066 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:05:07.225086 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:05:07.227292 (+ 2206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:05:07.230933 (+ 3641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:05:07.230939 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:05:07.230944 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:05:07.231405 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:05:07.231782 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:05:07.231785 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:05:07.231787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:05:07.231967 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:05:07.232597 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:05:07.232601 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:05:07.232714 (+ 113us) client_negotiation.cc:769] Sending connection context
0929 21:05:07.232901 (+ 187us) client_negotiation.cc:240] Negotiation successful
0929 21:05:07.233236 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 21:05:07.233819 22966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:07.220135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55942 (local address 127.0.97.60:42019)
0929 21:05:07.220332 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:07.220341 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:07.220357 (+ 16us) server_negotiation.cc:407] Connection header received
0929 21:05:07.220423 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:07.220430 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:07.220584 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:07.220698 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:07.221487 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:07.222497 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:07.223693 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:07.224103 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:07.224199 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:07.227478 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:05:07.227486 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:05:07.227491 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:05:07.227526 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:05:07.230787 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:07.231514 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:07.231520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:07.231523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:07.231661 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:07.232086 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:07.232092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:07.232095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:07.232478 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:05:07.232625 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:07.233014 (+ 389us) server_negotiation.cc:299] Negotiation successful
0929 21:05:07.233431 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":144,"server-negotiator.queue_time_us":89}
I20260929 21:05:07.237255 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:07.221793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45868 (local address 127.0.97.61:34517)
0929 21:05:07.221941 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:07.221944 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:07.221960 (+ 16us) server_negotiation.cc:407] Connection header received
0929 21:05:07.222092 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:07.222095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:07.222146 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:07.222219 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:07.223436 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:07.224226 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:07.225572 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:07.226258 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:07.226350 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:07.231283 (+ 4933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:05:07.231287 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:05:07.231290 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:05:07.231319 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:05:07.234606 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:07.235334 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:07.235339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:07.235343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:07.235471 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:07.235879 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:07.235884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:07.235888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:07.236151 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:05:07.236436 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:07.236710 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 21:05:07.236968 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 21:05:07.238978 22963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f422674aaaa349c1afd192f1eb5b564e: Bootstrap starting.
I20260929 21:05:07.241946 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 21:05:07.242725 22963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f422674aaaa349c1afd192f1eb5b564e: Neither blocks nor log segments found. Creating new log.
I20260929 21:05:07.243738 22963 log.cc:824] T 00000000000000000000000000000000 P f422674aaaa349c1afd192f1eb5b564e: Log is configured to *not* fsync() on all Append() calls
I20260929 21:05:07.246343 22963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f422674aaaa349c1afd192f1eb5b564e: No bootstrap required, opened a new log
I20260929 21:05:07.244503 22790 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a53e0908b41f4ff49f73f3a53c2dc0c2"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60460
I20260929 21:05:07.253208 22963 raft_consensus.cc:348] T 00000000000000000000000000000000 P f422674aaaa349c1afd192f1eb5b564e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53e0908b41f4ff49f73f3a53c2dc0c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38069 } } peers { permanent_uuid: "3ebc991527464415a4739345993e44e9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34517 } } peers { permanent_uuid: "f422674aaaa349c1afd192f1eb5b564e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42019 } }
I20260929 21:05:07.253531 22963 raft_consensus.cc:729] T 00000000000000000000000000000000 P f422674aaaa349c1afd192f1eb5b564e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f422674aaaa349c1afd192f1eb5b564e, State: Initialized, Role: FOLLOWER
I20260929 21:05:07.253957 22963 consensus_queue.cc:260] T 00000000000000000000000000000000 P f422674aaaa349c1afd192f1eb5b564e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53e0908b41f4ff49f73f3a53c2dc0c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38069 } } peers { permanent_uuid: "3ebc991527464415a4739345993e44e9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34517 } } peers { permanent_uuid: "f422674aaaa349c1afd192f1eb5b564e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42019 } }
I20260929 21:05:07.254912 22969 sys_catalog.cc:455] T 00000000000000000000000000000000 P f422674aaaa349c1afd192f1eb5b564e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53e0908b41f4ff49f73f3a53c2dc0c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38069 } } peers { permanent_uuid: "3ebc991527464415a4739345993e44e9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34517 } } peers { permanent_uuid: "f422674aaaa349c1afd192f1eb5b564e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42019 } } }
I20260929 21:05:07.255280 22969 sys_catalog.cc:458] T 00000000000000000000000000000000 P f422674aaaa349c1afd192f1eb5b564e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 21:05:07.255774 22963 sys_catalog.cc:564] T 00000000000000000000000000000000 P f422674aaaa349c1afd192f1eb5b564e [sys.catalog]: configured and running, proceeding with master startup.
W20260929 21:05:07.264910 22980 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f422674aaaa349c1afd192f1eb5b564e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 21:05:07.265039 22980 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 21:05:07.262751 22896 master_runner.cc:186] Error getting master registration for 127.0.97.62:38069: OK, instance_id { permanent_uuid: "a53e0908b41f4ff49f73f3a53c2dc0c2" instance_seqno: 1790715906161179 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 21:05:07.273728 22896 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 21:05:07.299316 22815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42019 } attempt: 4
Sep 29 21:05:07 dist-test-slave-1dms krb5kdc[22735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715906, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 21:05:07.316391 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:07.300086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42019 (local address 127.0.0.1:55948)
0929 21:05:07.300302 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:07.300320 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 21:05:07.300431 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:05:07.300854 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:05:07.300862 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:05:07.301389 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:05:07.301601 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:05:07.301617 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:05:07.303065 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:05:07.303070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:05:07.304531 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:05:07.304548 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:05:07.305591 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:05:07.305598 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:05:07.305811 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:07.306491 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:05:07.306515 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:05:07.309514 (+ 2999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:05:07.313363 (+ 3849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:05:07.313370 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:05:07.313374 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:05:07.313932 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:05:07.314398 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:05:07.314403 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:05:07.314407 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:05:07.314587 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:05:07.315301 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:05:07.315306 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:05:07.315493 (+ 187us) client_negotiation.cc:769] Sending connection context
0929 21:05:07.315657 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 21:05:07.315975 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 21:05:07.316404 22966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:07.300199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55948 (local address 127.0.97.60:42019)
0929 21:05:07.300416 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:07.300422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:07.300445 (+ 23us) server_negotiation.cc:407] Connection header received
0929 21:05:07.300564 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:07.300570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:07.300661 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:07.300785 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:07.301771 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:07.302917 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:07.304683 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:07.305437 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:07.305569 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:07.309739 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:05:07.309746 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:05:07.309751 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:05:07.309795 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:05:07.313221 (+ 3426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:07.314070 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:07.314075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:07.314085 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:07.314270 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:07.314704 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:07.314708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:07.314712 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:07.315166 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:05:07.315335 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:07.315770 (+ 435us) server_negotiation.cc:299] Negotiation successful
0929 21:05:07.316010 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 21:05:07.322582 22815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a53e0908b41f4ff49f73f3a53c2dc0c2: Bootstrap starting.
I20260929 21:05:07.326633 22815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a53e0908b41f4ff49f73f3a53c2dc0c2: Neither blocks nor log segments found. Creating new log.
I20260929 21:05:07.328006 22815 log.cc:824] T 00000000000000000000000000000000 P a53e0908b41f4ff49f73f3a53c2dc0c2: Log is configured to *not* fsync() on all Append() calls
I20260929 21:05:07.332211 22815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a53e0908b41f4ff49f73f3a53c2dc0c2: No bootstrap required, opened a new log
I20260929 21:05:07.340029 22815 raft_consensus.cc:348] T 00000000000000000000000000000000 P a53e0908b41f4ff49f73f3a53c2dc0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53e0908b41f4ff49f73f3a53c2dc0c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38069 } } peers { permanent_uuid: "3ebc991527464415a4739345993e44e9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34517 } } peers { permanent_uuid: "f422674aaaa349c1afd192f1eb5b564e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42019 } }
I20260929 21:05:07.340436 22815 raft_consensus.cc:729] T 00000000000000000000000000000000 P a53e0908b41f4ff49f73f3a53c2dc0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a53e0908b41f4ff49f73f3a53c2dc0c2, State: Initialized, Role: FOLLOWER
I20260929 21:05:07.340970 22815 consensus_queue.cc:260] T 00000000000000000000000000000000 P a53e0908b41f4ff49f73f3a53c2dc0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53e0908b41f4ff49f73f3a53c2dc0c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38069 } } peers { permanent_uuid: "3ebc991527464415a4739345993e44e9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34517 } } peers { permanent_uuid: "f422674aaaa349c1afd192f1eb5b564e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42019 } }
I20260929 21:05:07.342243 22983 sys_catalog.cc:455] T 00000000000000000000000000000000 P a53e0908b41f4ff49f73f3a53c2dc0c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53e0908b41f4ff49f73f3a53c2dc0c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38069 } } peers { permanent_uuid: "3ebc991527464415a4739345993e44e9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34517 } } peers { permanent_uuid: "f422674aaaa349c1afd192f1eb5b564e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42019 } } }
I20260929 21:05:07.342787 22983 sys_catalog.cc:458] T 00000000000000000000000000000000 P a53e0908b41f4ff49f73f3a53c2dc0c2 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 21:05:07.343410 22815 sys_catalog.cc:564] T 00000000000000000000000000000000 P a53e0908b41f4ff49f73f3a53c2dc0c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:05:07.343510 22790 raft_consensus.cc:482] T 00000000000000000000000000000000 P a53e0908b41f4ff49f73f3a53c2dc0c2 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 21:05:07.343688 22790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a53e0908b41f4ff49f73f3a53c2dc0c2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:05:07.347721 22790 raft_consensus.cc:504] T 00000000000000000000000000000000 P a53e0908b41f4ff49f73f3a53c2dc0c2 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53e0908b41f4ff49f73f3a53c2dc0c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38069 } } peers { permanent_uuid: "3ebc991527464415a4739345993e44e9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34517 } } peers { permanent_uuid: "f422674aaaa349c1afd192f1eb5b564e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42019 } }
I20260929 21:05:07.351234 22938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a53e0908b41f4ff49f73f3a53c2dc0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f422674aaaa349c1afd192f1eb5b564e"
I20260929 21:05:07.351745 22938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f422674aaaa349c1afd192f1eb5b564e [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:05:07.352155 22790 leader_election.cc:288] T 00000000000000000000000000000000 P a53e0908b41f4ff49f73f3a53c2dc0c2 [CANDIDATE]: Term 1 election: Requested vote from peers 3ebc991527464415a4739345993e44e9 (127.0.97.61:34517), f422674aaaa349c1afd192f1eb5b564e (127.0.97.60:42019)
I20260929 21:05:07.355365 22938 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f422674aaaa349c1afd192f1eb5b564e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a53e0908b41f4ff49f73f3a53c2dc0c2 in term 1.
W20260929 21:05:07.355836 22994 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a53e0908b41f4ff49f73f3a53c2dc0c2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 21:05:07.356012 22994 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 21:05:07.356419 22767 leader_election.cc:302] T 00000000000000000000000000000000 P a53e0908b41f4ff49f73f3a53c2dc0c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a53e0908b41f4ff49f73f3a53c2dc0c2, f422674aaaa349c1afd192f1eb5b564e; no voters:
I20260929 21:05:07.361933 22983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a53e0908b41f4ff49f73f3a53c2dc0c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:05:07.353003 22748 master_runner.cc:186] Error getting master registration for 127.0.97.61:34517: OK, instance_id { permanent_uuid: "3ebc991527464415a4739345993e44e9" instance_seqno: 1790715906664626 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 21:05:07.365939 22748 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 21:05:07.367141 22983 raft_consensus.cc:686] T 00000000000000000000000000000000 P a53e0908b41f4ff49f73f3a53c2dc0c2 [term 1 LEADER]: Becoming Leader. State: Replica: a53e0908b41f4ff49f73f3a53c2dc0c2, State: Running, Role: LEADER
I20260929 21:05:07.367702 22983 consensus_queue.cc:237] T 00000000000000000000000000000000 P a53e0908b41f4ff49f73f3a53c2dc0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53e0908b41f4ff49f73f3a53c2dc0c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38069 } } peers { permanent_uuid: "3ebc991527464415a4739345993e44e9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34517 } } peers { permanent_uuid: "f422674aaaa349c1afd192f1eb5b564e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42019 } }
I20260929 21:05:07.377914 22997 sys_catalog.cc:455] T 00000000000000000000000000000000 P a53e0908b41f4ff49f73f3a53c2dc0c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a53e0908b41f4ff49f73f3a53c2dc0c2. Latest consensus state: current_term: 1 leader_uuid: "a53e0908b41f4ff49f73f3a53c2dc0c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53e0908b41f4ff49f73f3a53c2dc0c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38069 } } peers { permanent_uuid: "3ebc991527464415a4739345993e44e9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34517 } } peers { permanent_uuid: "f422674aaaa349c1afd192f1eb5b564e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42019 } } }
I20260929 21:05:07.378326 22997 sys_catalog.cc:458] T 00000000000000000000000000000000 P a53e0908b41f4ff49f73f3a53c2dc0c2 [sys.catalog]: This master's current role is: LEADER
I20260929 21:05:07.387212 23002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:05:07.389583 23002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:05:07.392319 22889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42019 } attempt: 5
W20260929 21:05:07.400815 22769 tablet.cc:2369] T 00000000000000000000000000000000 P a53e0908b41f4ff49f73f3a53c2dc0c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 21:05:07.405508 22917 tablet.cc:2369] T 00000000000000000000000000000000 P f422674aaaa349c1afd192f1eb5b564e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 21:05:07.420487 22938 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f422674aaaa349c1afd192f1eb5b564e [term 1 FOLLOWER]: Refusing update from remote peer a53e0908b41f4ff49f73f3a53c2dc0c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:05:07.421667 22997 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a53e0908b41f4ff49f73f3a53c2dc0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f422674aaaa349c1afd192f1eb5b564e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 29 21:05:07 dist-test-slave-1dms krb5kdc[22735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715906, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 21:05:07.437668 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:07.383650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60476 (local address 127.0.97.62:38069)
0929 21:05:07.383821 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:07.383827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:07.389787 (+ 5960us) server_negotiation.cc:407] Connection header received
0929 21:05:07.390061 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:07.390066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:07.390157 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:07.390235 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:07.400866 (+ 10631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:07.402346 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:07.408795 (+ 6449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:07.409687 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:07.409791 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:07.421321 (+ 11530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:05:07.421328 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:05:07.421332 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:05:07.421381 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:05:07.425534 (+ 4153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:07.431421 (+ 5887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:07.431431 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:07.431438 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:07.431686 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:07.433776 (+ 2090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:07.433782 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:07.433785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:07.434166 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:05:07.434427 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:07.436883 (+ 2456us) server_negotiation.cc:299] Negotiation successful
0929 21:05:07.437170 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 21:05:07.451786 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:07.392892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42019 (local address 127.0.0.1:55954)
0929 21:05:07.393414 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:07.393437 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 21:05:07.393560 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:05:07.415536 (+ 21976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:05:07.415542 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:05:07.416130 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:05:07.416351 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:05:07.416366 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:05:07.428543 (+ 12177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:05:07.428549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:05:07.430077 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:05:07.430095 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:05:07.432305 (+ 2210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:05:07.432312 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:05:07.432551 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:07.433234 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:05:07.433260 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:05:07.436124 (+ 2864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:05:07.446538 (+ 10414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:05:07.446547 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:05:07.446552 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:05:07.447175 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:05:07.447713 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:05:07.447720 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:05:07.447726 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:05:07.447909 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:05:07.450653 (+ 2744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:05:07.450673 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:05:07.450852 (+ 179us) client_negotiation.cc:769] Sending connection context
0929 21:05:07.451008 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 21:05:07.451323 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384}
I20260929 21:05:07.453861 23006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:07.393874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55954 (local address 127.0.97.60:42019)
0929 21:05:07.415105 (+ 21231us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:07.415110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:07.415131 (+ 21us) server_negotiation.cc:407] Connection header received
0929 21:05:07.415170 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:07.415177 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:07.415258 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:07.416524 (+ 1266us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:07.416877 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:07.418184 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:07.430510 (+ 12326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:07.431235 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:07.436316 (+ 5081us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:07.436583 (+ 267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:05:07.436591 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:05:07.436598 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:05:07.436632 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:05:07.442564 (+ 5932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:07.447355 (+ 4791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:07.447362 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:07.447367 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:07.447550 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:07.448069 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:07.448076 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:07.448080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:07.448465 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:05:07.452218 (+ 3753us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:07.452354 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 21:05:07.452600 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21050,"thread_start_us":21447,"threads_started":1}
I20260929 21:05:07.460776 22889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ebc991527464415a4739345993e44e9: Bootstrap starting.
I20260929 21:05:07.465343 22889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ebc991527464415a4739345993e44e9: Neither blocks nor log segments found. Creating new log.
I20260929 21:05:07.466845 22889 log.cc:824] T 00000000000000000000000000000000 P 3ebc991527464415a4739345993e44e9: Log is configured to *not* fsync() on all Append() calls
I20260929 21:05:07.469539 22969 sys_catalog.cc:455] T 00000000000000000000000000000000 P f422674aaaa349c1afd192f1eb5b564e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a53e0908b41f4ff49f73f3a53c2dc0c2. Latest consensus state: current_term: 1 leader_uuid: "a53e0908b41f4ff49f73f3a53c2dc0c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53e0908b41f4ff49f73f3a53c2dc0c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38069 } } peers { permanent_uuid: "3ebc991527464415a4739345993e44e9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34517 } } peers { permanent_uuid: "f422674aaaa349c1afd192f1eb5b564e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42019 } } }
I20260929 21:05:07.469857 22969 sys_catalog.cc:458] T 00000000000000000000000000000000 P f422674aaaa349c1afd192f1eb5b564e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 21:05:07.470520 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:07.390092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45872 (local address 127.0.97.61:34517)
0929 21:05:07.390529 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:07.390538 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:07.408426 (+ 17888us) server_negotiation.cc:407] Connection header received
0929 21:05:07.412011 (+ 3585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:07.412017 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:07.412107 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:07.412181 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:07.426991 (+ 14810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:07.428088 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:07.442433 (+ 14345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:07.443253 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:07.443375 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:07.454425 (+ 11050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:05:07.454432 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:05:07.454440 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:05:07.454478 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:05:07.458555 (+ 4077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:07.465030 (+ 6475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:07.465036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:07.465039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:07.465219 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:07.466716 (+ 1497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:07.466722 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:07.466725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:07.467132 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:05:07.467638 (+ 506us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:07.469637 (+ 1999us) server_negotiation.cc:299] Negotiation successful
0929 21:05:07.470014 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291}
I20260929 21:05:07.470844 22966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:07.392569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55952 (local address 127.0.97.60:42019)
0929 21:05:07.393246 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:07.393255 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:07.411294 (+ 18039us) server_negotiation.cc:407] Connection header received
0929 21:05:07.414873 (+ 3579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:07.414884 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:07.414995 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:07.415071 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:07.419728 (+ 4657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:07.420776 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:07.438566 (+ 17790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:07.439324 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:07.439428 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:07.458814 (+ 19386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:05:07.458822 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:05:07.458827 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:05:07.458869 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:05:07.462635 (+ 3766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:07.465431 (+ 2796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:07.465437 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:07.465442 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:07.465583 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:07.467817 (+ 2234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:07.467872 (+ 55us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:07.467878 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:07.468512 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:05:07.470112 (+ 1600us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:07.470264 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 21:05:07.470518 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498}
I20260929 21:05:07.471917 22889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ebc991527464415a4739345993e44e9: No bootstrap required, opened a new log
I20260929 21:05:07.484709 23007 mvcc.cc:205] Tried to move back new op lower bound from 7334772356758487040 to 7334772356583804928. Current Snapshot: MvccSnapshot[applied={T|T < 7334772356758487040}]
I20260929 21:05:07.490820 22889 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ebc991527464415a4739345993e44e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53e0908b41f4ff49f73f3a53c2dc0c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38069 } } peers { permanent_uuid: "3ebc991527464415a4739345993e44e9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34517 } } peers { permanent_uuid: "f422674aaaa349c1afd192f1eb5b564e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42019 } }
I20260929 21:05:07.492250 22889 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ebc991527464415a4739345993e44e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ebc991527464415a4739345993e44e9, State: Initialized, Role: FOLLOWER
I20260929 21:05:07.493393 22889 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ebc991527464415a4739345993e44e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53e0908b41f4ff49f73f3a53c2dc0c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38069 } } peers { permanent_uuid: "3ebc991527464415a4739345993e44e9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34517 } } peers { permanent_uuid: "f422674aaaa349c1afd192f1eb5b564e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42019 } }
I20260929 21:05:07.499500 22983 sys_catalog.cc:455] T 00000000000000000000000000000000 P a53e0908b41f4ff49f73f3a53c2dc0c2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a53e0908b41f4ff49f73f3a53c2dc0c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53e0908b41f4ff49f73f3a53c2dc0c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38069 } } peers { permanent_uuid: "3ebc991527464415a4739345993e44e9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34517 } } peers { permanent_uuid: "f422674aaaa349c1afd192f1eb5b564e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42019 } } }
I20260929 21:05:07.499889 22983 sys_catalog.cc:458] T 00000000000000000000000000000000 P a53e0908b41f4ff49f73f3a53c2dc0c2 [sys.catalog]: This master's current role is: LEADER
I20260929 21:05:07.500727 22969 sys_catalog.cc:455] T 00000000000000000000000000000000 P f422674aaaa349c1afd192f1eb5b564e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a53e0908b41f4ff49f73f3a53c2dc0c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53e0908b41f4ff49f73f3a53c2dc0c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38069 } } peers { permanent_uuid: "3ebc991527464415a4739345993e44e9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34517 } } peers { permanent_uuid: "f422674aaaa349c1afd192f1eb5b564e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42019 } } }
I20260929 21:05:07.501013 22969 sys_catalog.cc:458] T 00000000000000000000000000000000 P f422674aaaa349c1afd192f1eb5b564e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 21:05:07.501415 23013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ebc991527464415a4739345993e44e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53e0908b41f4ff49f73f3a53c2dc0c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38069 } } peers { permanent_uuid: "3ebc991527464415a4739345993e44e9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34517 } } peers { permanent_uuid: "f422674aaaa349c1afd192f1eb5b564e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42019 } } }
I20260929 21:05:07.503038 23013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ebc991527464415a4739345993e44e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 21:05:07.504755 23002 catalog_manager.cc:1347] Generated new cluster ID: 7e2cb4ca516c4b89a302cc77d92cd6c4
I20260929 21:05:07.504881 23002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:05:07.506472 22889 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ebc991527464415a4739345993e44e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:05:07.509919 22863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ebc991527464415a4739345993e44e9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:05:07.512990 22864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a53e0908b41f4ff49f73f3a53c2dc0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3ebc991527464415a4739345993e44e9"
I20260929 21:05:07.523836 22863 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3ebc991527464415a4739345993e44e9 [term 1 FOLLOWER]: Refusing update from remote peer a53e0908b41f4ff49f73f3a53c2dc0c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:05:07.528481 22864 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3ebc991527464415a4739345993e44e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a53e0908b41f4ff49f73f3a53c2dc0c2 in term 1.
I20260929 21:05:07.529438 22997 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a53e0908b41f4ff49f73f3a53c2dc0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ebc991527464415a4739345993e44e9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:05:07.545420 23002 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:05:07.549912 23002 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:05:07.578503 23002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a53e0908b41f4ff49f73f3a53c2dc0c2: Generated new TSK 0
I20260929 21:05:07.579664 23002 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 21:05:07.585984 23023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ebc991527464415a4739345993e44e9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a53e0908b41f4ff49f73f3a53c2dc0c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53e0908b41f4ff49f73f3a53c2dc0c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38069 } } peers { permanent_uuid: "3ebc991527464415a4739345993e44e9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34517 } } peers { permanent_uuid: "f422674aaaa349c1afd192f1eb5b564e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42019 } } }
I20260929 21:05:07.586455 23023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ebc991527464415a4739345993e44e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 21:05:07.586454 22997 sys_catalog.cc:455] T 00000000000000000000000000000000 P a53e0908b41f4ff49f73f3a53c2dc0c2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a53e0908b41f4ff49f73f3a53c2dc0c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53e0908b41f4ff49f73f3a53c2dc0c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38069 } } peers { permanent_uuid: "3ebc991527464415a4739345993e44e9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34517 } } peers { permanent_uuid: "f422674aaaa349c1afd192f1eb5b564e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42019 } } }
I20260929 21:05:07.586730 22997 sys_catalog.cc:458] T 00000000000000000000000000000000 P a53e0908b41f4ff49f73f3a53c2dc0c2 [sys.catalog]: This master's current role is: LEADER
I20260929 21:05:07.587335 23013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ebc991527464415a4739345993e44e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a53e0908b41f4ff49f73f3a53c2dc0c2. Latest consensus state: current_term: 1 leader_uuid: "a53e0908b41f4ff49f73f3a53c2dc0c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53e0908b41f4ff49f73f3a53c2dc0c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38069 } } peers { permanent_uuid: "3ebc991527464415a4739345993e44e9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34517 } } peers { permanent_uuid: "f422674aaaa349c1afd192f1eb5b564e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42019 } } }
I20260929 21:05:07.587600 23013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ebc991527464415a4739345993e44e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 21:05:07.595767 23028 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260929 21:05:07.670966 22817 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:05:07.661718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60480 (local address 127.0.97.62:38069)
0929 21:05:07.661933 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:07.661939 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:07.668924 (+ 6985us) server_negotiation.cc:407] Connection header received
0929 21:05:07.669414 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:07.669422 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:07.669524 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:07.669607 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:07.670715 (+ 1108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60480: BlockingRecv error: recv got EOF from 127.0.0.1:60480 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260929 21:05:07.673134 22966 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:05:07.661802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55964 (local address 127.0.97.60:42019)
0929 21:05:07.661966 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:07.661972 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:07.671555 (+ 9583us) server_negotiation.cc:407] Connection header received
0929 21:05:07.671820 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:07.671829 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:07.671916 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:07.671993 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:07.672861 (+ 868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55964: BlockingRecv error: recv got EOF from 127.0.0.1:55964 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 21:05:07.684372 22893 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:05:07.682124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45882 (local address 127.0.97.61:34517)
0929 21:05:07.682316 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:07.682322 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:07.682793 (+ 471us) server_negotiation.cc:407] Connection header received
0929 21:05:07.683023 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:07.683029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:07.683099 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:07.683163 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:07.684150 (+ 987us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45882: BlockingRecv error: recv got EOF from 127.0.0.1:45882 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 21:05:07.718039 22893 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:05:07.715799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45896 (local address 127.0.97.61:34517)
0929 21:05:07.716019 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:07.716029 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:07.716527 (+ 498us) server_negotiation.cc:407] Connection header received
0929 21:05:07.716794 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:07.716798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:07.716965 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:07.717079 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:07.717783 (+ 704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45896: BlockingRecv error: recv got EOF from 127.0.0.1:45896 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260929 21:05:07.728816 22893 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:05:07.726611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45912 (local address 127.0.97.61:34517)
0929 21:05:07.726855 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:07.726864 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:07.727372 (+ 508us) server_negotiation.cc:407] Connection header received
0929 21:05:07.727569 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:07.727574 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:07.727647 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:07.727751 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:07.728616 (+ 865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45912: BlockingRecv error: recv got EOF from 127.0.0.1:45912 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
W20260929 21:05:07.735581 22893 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:05:07.733719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45918 (local address 127.0.97.61:34517)
0929 21:05:07.733895 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:07.733913 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:07.734313 (+ 400us) server_negotiation.cc:407] Connection header received
0929 21:05:07.734557 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:07.734561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:07.734620 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:07.734724 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:07.735348 (+ 624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45918: BlockingRecv error: recv got EOF from 127.0.0.1:45918 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 21:05:07.744442 22893 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:05:07.742324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45930 (local address 127.0.97.61:34517)
0929 21:05:07.742544 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:07.742553 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:07.743034 (+ 481us) server_negotiation.cc:407] Connection header received
0929 21:05:07.743224 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:07.743231 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:07.743298 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:07.743411 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:07.744211 (+ 800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45930: BlockingRecv error: recv got EOF from 127.0.0.1:45930 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260929 21:05:07.757927 22893 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:05:07.755728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45942 (local address 127.0.97.61:34517)
0929 21:05:07.755942 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:07.755951 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:07.756446 (+ 495us) server_negotiation.cc:407] Connection header received
0929 21:05:07.756675 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:07.756680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:07.756756 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:07.756882 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:07.757695 (+ 813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45942: BlockingRecv error: recv got EOF from 127.0.0.1:45942 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 21:05:07.779136 22893 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:05:07.777007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45952 (local address 127.0.97.61:34517)
0929 21:05:07.777239 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:07.777248 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:07.777815 (+ 567us) server_negotiation.cc:407] Connection header received
0929 21:05:07.778053 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:07.778057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:07.778136 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:07.778243 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:07.778904 (+ 661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45952: BlockingRecv error: recv got EOF from 127.0.0.1:45952 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 21:05:07.816673 22893 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:05:07.814483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45966 (local address 127.0.97.61:34517)
0929 21:05:07.814712 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:07.814721 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:07.815233 (+ 512us) server_negotiation.cc:407] Connection header received
0929 21:05:07.815469 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:07.815473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:07.815545 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:07.815654 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:07.816388 (+ 734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45966: BlockingRecv error: recv got EOF from 127.0.0.1:45966 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 21:05:07.886044 22893 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:05:07.884096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45982 (local address 127.0.97.61:34517)
0929 21:05:07.884310 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:07.884319 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:07.884724 (+ 405us) server_negotiation.cc:407] Connection header received
0929 21:05:07.884876 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:07.884883 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:07.884980 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:07.885103 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:07.885827 (+ 724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45982: BlockingRecv error: recv got EOF from 127.0.0.1:45982 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260929 21:05:08.020287 22893 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:05:08.018263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45992 (local address 127.0.97.61:34517)
0929 21:05:08.018496 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:08.018505 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:08.018831 (+ 326us) server_negotiation.cc:407] Connection header received
0929 21:05:08.019066 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:08.019070 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:08.019152 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:08.019265 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:08.020014 (+ 749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45992: BlockingRecv error: recv got EOF from 127.0.0.1:45992 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 21:05:08.267056 22980 catalog_manager.cc:1259] Loaded cluster ID: 7e2cb4ca516c4b89a302cc77d92cd6c4
I20260929 21:05:08.267160 22980 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f422674aaaa349c1afd192f1eb5b564e: loading cluster ID for follower catalog manager: success
I20260929 21:05:08.269582 22980 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f422674aaaa349c1afd192f1eb5b564e: acquiring CA information for follower catalog manager: success
I20260929 21:05:08.271342 22980 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f422674aaaa349c1afd192f1eb5b564e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 21:05:08.281556 22893 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:05:08.279780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46000 (local address 127.0.97.61:34517)
0929 21:05:08.280020 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:08.280029 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:08.280167 (+ 138us) server_negotiation.cc:407] Connection header received
0929 21:05:08.280329 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:08.280334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:08.280420 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:08.280508 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:08.281306 (+ 798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46000: BlockingRecv error: recv got EOF from 127.0.0.1:46000 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 21:05:08.600207 23028 catalog_manager.cc:1259] Loaded cluster ID: 7e2cb4ca516c4b89a302cc77d92cd6c4
I20260929 21:05:08.600323 23028 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3ebc991527464415a4739345993e44e9: loading cluster ID for follower catalog manager: success
I20260929 21:05:08.602869 23028 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3ebc991527464415a4739345993e44e9: acquiring CA information for follower catalog manager: success
I20260929 21:05:08.604756 23028 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3ebc991527464415a4739345993e44e9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 21:05:08.814775 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:08.808314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47648 (local address 127.0.97.61:34517)
0929 21:05:08.809207 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:08.809212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:08.809228 (+ 16us) server_negotiation.cc:407] Connection header received
0929 21:05:08.809293 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:08.809297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:08.809439 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:08.809586 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:05:08.810102 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:08.810932 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:08.812321 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:08.812782 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:08.812998 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:08.813514 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:05:08.814049 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:05:08.814210 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:08.814341 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 21:05:08.814433 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":546,"threads_started":1}
I20260929 21:05:08.833467 23101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:08.821205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33380 (local address 127.0.97.60:42019)
0929 21:05:08.828119 (+ 6914us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:08.828123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:08.828140 (+ 17us) server_negotiation.cc:407] Connection header received
0929 21:05:08.828228 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:08.828232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:08.828444 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:08.828619 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:05:08.829097 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:08.829945 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:08.831318 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:08.831708 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:08.831838 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:08.832345 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:05:08.832825 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:05:08.833042 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:08.833104 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 21:05:08.833182 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6815,"thread_start_us":6722,"threads_started":1}
I20260929 21:05:08.860571 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:08.839544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35362 (local address 127.0.97.62:38069)
0929 21:05:08.851983 (+ 12439us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:08.851992 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:08.852526 (+ 534us) server_negotiation.cc:407] Connection header received
0929 21:05:08.852655 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:08.852663 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:08.852919 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:08.853780 (+ 861us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:05:08.853962 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:08.855100 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:08.857580 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:08.858238 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:08.858859 (+ 621us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:08.859062 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:05:08.859517 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:05:08.860087 (+ 570us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:08.860144 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 21:05:08.860241 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12257,"thread_start_us":12131,"threads_started":1}
I20260929 21:05:08.865208 23101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:08.839699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33384 (local address 127.0.97.60:42019)
0929 21:05:08.839863 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:08.839872 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:08.856245 (+ 16373us) server_negotiation.cc:407] Connection header received
0929 21:05:08.856454 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:08.856463 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:08.856593 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:08.856694 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:05:08.858306 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:08.859459 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:08.861819 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:08.862427 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:08.862513 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:08.864374 (+ 1861us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:05:08.864520 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:05:08.864599 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:08.864783 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 21:05:08.864893 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 21:05:08.865618 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:08.852648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47664 (local address 127.0.97.61:34517)
0929 21:05:08.852807 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:08.852812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:08.857546 (+ 4734us) server_negotiation.cc:407] Connection header received
0929 21:05:08.857732 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:08.857738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:08.857824 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:08.857942 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:05:08.858543 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:08.859692 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:08.862321 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:08.863066 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:08.863159 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:08.863881 (+ 722us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:05:08.864018 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:05:08.864125 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:08.865209 (+ 1084us) server_negotiation.cc:299] Negotiation successful
0929 21:05:08.865298 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 21:05:08.865690 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 22748
I20260929 21:05:08.867347 22810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:05:08.979921 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 22748
I20260929 21:05:09.000522 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 22822
I20260929 21:05:09.001664 22884 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:05:09.109072 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 22822
I20260929 21:05:09.129388 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 22896
I20260929 21:05:09.130527 22958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:05:09.237587 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 22896
2026-09-29T21:05:09Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4861 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 21:05:09 dist-test-slave-1dms krb5kdc[23115](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 21:05:09 dist-test-slave-1dms krb5kdc[23115](info): set up 2 sockets
Sep 29 21:05:09 dist-test-slave-1dms krb5kdc[23115](info): commencing operation
krb5kdc: starting...
W20260929 21:05:10.337354 388 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/krb5kdc/test-user.keytab.
Sep 29 21:05:10 dist-test-slave-1dms krb5kdc[23115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T21:05:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T21:05:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 21:05:10.580097 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46339
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46339,127.0.97.61:37807,127.0.97.60:32997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 21:05:10.901749 23128 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 21:05:10.902093 23128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:05:10.902222 23128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:05:10.902351 23128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:05:10.909456 23128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:05:10.909538 23128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:05:10.909633 23128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:05:10.909731 23128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:05:10.909823 23128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:05:10.919231 23128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46339,127.0.97.61:37807,127.0.97.60:32997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:05:10.920511 23128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:05:10.921801 23128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 21:05:10.937345 23133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:05:10.937407 23134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:05:10.937433 23136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:05:10.938143 23128 server_base.cc:1034] running on GCE node
I20260929 21:05:10.938835 23128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:05:10.940132 23128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:05:10.941349 23128 hybrid_clock.cc:648] HybridClock initialized: now 1790715910941293 us; error 31 us; skew 500 ppm
I20260929 21:05:10.941681 23128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:05:10 dist-test-slave-1dms krb5kdc[23115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:05:10.944671 23128 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 21:05:10.946130 23128 webserver.cc:460] Webserver started at http://127.0.97.62:45765/ using document root <none> and password file <none>
I20260929 21:05:10.946808 23128 fs_manager.cc:359] Metadata directory not provided
I20260929 21:05:10.946923 23128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:05:10.947242 23128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:05:10.950188 23128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-0/data/instance:
uuid: "c71f061c92ea49be9f1289e71ac10917"
format_stamp: "Formatted at 2026-09-29 21:05:10 on dist-test-slave-1dms"
server_key: "6c87f57c34d2003dd9b19845861e0571"
server_key_iv: "24f6b202633ceadf80c8e8470218307e"
server_key_version: "encryptionkey@0"
I20260929 21:05:10.950963 23128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-0/wal/instance:
uuid: "c71f061c92ea49be9f1289e71ac10917"
format_stamp: "Formatted at 2026-09-29 21:05:10 on dist-test-slave-1dms"
server_key: "6c87f57c34d2003dd9b19845861e0571"
server_key_iv: "24f6b202633ceadf80c8e8470218307e"
server_key_version: "encryptionkey@0"
I20260929 21:05:10.955788 23128 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260929 21:05:10.959129 23143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:05:10.960439 23128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 21:05:10.960625 23128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-0/wal
uuid: "c71f061c92ea49be9f1289e71ac10917"
format_stamp: "Formatted at 2026-09-29 21:05:10 on dist-test-slave-1dms"
server_key: "6c87f57c34d2003dd9b19845861e0571"
server_key_iv: "24f6b202633ceadf80c8e8470218307e"
server_key_version: "encryptionkey@0"
I20260929 21:05:10.960845 23128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:05:10.996748 23128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:05:11.001137 23128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:05:11.001402 23128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:05:11.020203 23128 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46339
I20260929 21:05:11.020226 23194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46339 every 8 connection(s)
I20260929 21:05:11.021610 23128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-0/data/info.pb
I20260929 21:05:11.022934 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 23128
I20260929 21:05:11.023123 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-0/wal/instance
I20260929 21:05:11.023682 388 external_mini_cluster.cc:1428] Setting key 46addf561ef82a17f39bb26fac342f5b
I20260929 21:05:11.026185 23195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:05:11.035805 23195 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46339 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Sep 29 21:05:11 dist-test-slave-1dms krb5kdc[23115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:05:11.059803 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:11.039063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46339 (local address 127.0.0.1:58858)
0929 21:05:11.039949 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:11.039995 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 21:05:11.041944 (+ 1949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:05:11.042760 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:05:11.042767 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:05:11.043456 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:05:11.043774 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:05:11.043798 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:05:11.045367 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:05:11.045374 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:05:11.046524 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:05:11.046535 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:05:11.047459 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:05:11.047467 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:05:11.047631 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:11.048287 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:05:11.048311 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:05:11.052818 (+ 4507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:05:11.056074 (+ 3256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:05:11.056086 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:05:11.056098 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:05:11.056484 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:05:11.056902 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:05:11.056908 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:05:11.056912 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:05:11.057081 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:05:11.057799 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:05:11.057807 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:05:11.058019 (+ 212us) client_negotiation.cc:769] Sending connection context
0929 21:05:11.058336 (+ 317us) client_negotiation.cc:240] Negotiation successful
0929 21:05:11.058924 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":66,"thread_start_us":396,"threads_started":1}
I20260929 21:05:11.059836 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:11.039071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58858 (local address 127.0.97.62:46339)
0929 21:05:11.040763 (+ 1692us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:11.040772 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:11.040817 (+ 45us) server_negotiation.cc:407] Connection header received
0929 21:05:11.042310 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:11.042315 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:11.042541 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:11.042746 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:11.043987 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:11.045213 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:11.046662 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:11.047223 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:11.047435 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:11.053097 (+ 5662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:05:11.053105 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:05:11.053116 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:05:11.053145 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:05:11.055931 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:11.056612 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:11.056620 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:11.056623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:11.056768 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:11.057187 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:11.057195 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:11.057198 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:11.057663 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:05:11.057879 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:11.058461 (+ 582us) server_negotiation.cc:299] Negotiation successful
0929 21:05:11.058731 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1495,"thread_start_us":1250,"threads_started":1}
I20260929 21:05:11.062664 23195 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37807 } has no permanent_uuid. Determining permanent_uuid...
I20260929 21:05:11.063850 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:11.063443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37807 (local address 127.0.0.1:54476)
0929 21:05:11.063594 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:11.063743 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 21:05:11.064100 23145 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37807: connect: Connection refused (error 111)
I20260929 21:05:11.065632 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:11.065314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37807 (local address 127.0.0.1:54490)
0929 21:05:11.065459 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:11.065518 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 21:05:11.065896 23195 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37807: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37807: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260929 21:05:11.080542 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:37807
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46339,127.0.97.61:37807,127.0.97.60:32997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 21:05:11.115226 23195 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37807 } attempt: 1
I20260929 21:05:11.116453 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:11.115894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37807 (local address 127.0.0.1:54492)
0929 21:05:11.116155 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:11.116264 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260929 21:05:11.117584 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:11.117176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37807 (local address 127.0.0.1:54504)
0929 21:05:11.117324 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:11.117426 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 21:05:11.117787 23195 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37807: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37807: connect: Connection refused (error 111)
I20260929 21:05:11.186054 23195 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37807 } attempt: 2
I20260929 21:05:11.187352 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:11.186794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37807 (local address 127.0.0.1:54516)
0929 21:05:11.187064 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:11.187184 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 21:05:11.188437 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:11.188028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37807 (local address 127.0.0.1:54524)
0929 21:05:11.188172 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:11.188266 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 21:05:11.188629 23195 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37807: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37807: connect: Connection refused (error 111)
I20260929 21:05:11.279913 23195 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37807 } attempt: 3
I20260929 21:05:11.281226 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:11.280669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37807 (local address 127.0.0.1:54540)
0929 21:05:11.280940 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:11.281060 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260929 21:05:11.282338 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:11.281909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37807 (local address 127.0.0.1:54546)
0929 21:05:11.282068 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:11.282168 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 21:05:11.282752 23195 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37807: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37807: connect: Connection refused (error 111)
W20260929 21:05:11.401345 23202 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 21:05:11.401633 23202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:05:11.401700 23202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:05:11.401782 23202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:05:11.408902 23202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:05:11.408987 23202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:05:11.409044 23202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:05:11.409094 23202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:05:11.409147 23202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:05:11.418716 23202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46339,127.0.97.61:37807,127.0.97.60:32997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:37807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:05:11.419909 23202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:05:11.421288 23202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 21:05:11.426010 23195 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37807 } attempt: 4
I20260929 21:05:11.427428 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:11.426774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37807 (local address 127.0.0.1:54562)
0929 21:05:11.427158 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:11.427260 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192}
I20260929 21:05:11.428627 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:11.428120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37807 (local address 127.0.0.1:54568)
0929 21:05:11.428290 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:11.428401 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65,"thread_start_us":295,"threads_started":1}
W20260929 21:05:11.428933 23195 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37807: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37807: connect: Connection refused (error 111)
W20260929 21:05:11.433229 23207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:05:11.433346 23210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:05:11.433372 23208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:05:11.434384 23202 server_base.cc:1034] running on GCE node
I20260929 21:05:11.435174 23202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:05:11.436489 23202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:05:11.437741 23202 hybrid_clock.cc:648] HybridClock initialized: now 1790715911437670 us; error 40 us; skew 500 ppm
I20260929 21:05:11.438073 23202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:05:11 dist-test-slave-1dms krb5kdc[23115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:05:11.441182 23202 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260929 21:05:11.442512 23202 webserver.cc:460] Webserver started at http://127.0.97.61:38407/ using document root <none> and password file <none>
I20260929 21:05:11.443145 23202 fs_manager.cc:359] Metadata directory not provided
I20260929 21:05:11.443228 23202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:05:11.443492 23202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:05:11.446353 23202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-1/data/instance:
uuid: "5c122f501d7f4faca40a9808b28e7cd5"
format_stamp: "Formatted at 2026-09-29 21:05:11 on dist-test-slave-1dms"
server_key: "607a9c8a9576793d50fbc16224b453d6"
server_key_iv: "7d6df8d6481e3f22e162c28c9e0f61e2"
server_key_version: "encryptionkey@0"
I20260929 21:05:11.447077 23202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-1/wal/instance:
uuid: "5c122f501d7f4faca40a9808b28e7cd5"
format_stamp: "Formatted at 2026-09-29 21:05:11 on dist-test-slave-1dms"
server_key: "607a9c8a9576793d50fbc16224b453d6"
server_key_iv: "7d6df8d6481e3f22e162c28c9e0f61e2"
server_key_version: "encryptionkey@0"
I20260929 21:05:11.451995 23202 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 21:05:11.455214 23218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:05:11.456499 23202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 21:05:11.456682 23202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-1/wal
uuid: "5c122f501d7f4faca40a9808b28e7cd5"
format_stamp: "Formatted at 2026-09-29 21:05:11 on dist-test-slave-1dms"
server_key: "607a9c8a9576793d50fbc16224b453d6"
server_key_iv: "7d6df8d6481e3f22e162c28c9e0f61e2"
server_key_version: "encryptionkey@0"
I20260929 21:05:11.456923 23202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:05:11.473155 23202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:05:11.477365 23202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:05:11.477653 23202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:05:11.495352 23202 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:37807
I20260929 21:05:11.495370 23269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:37807 every 8 connection(s)
I20260929 21:05:11.496610 23202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-1/data/info.pb
I20260929 21:05:11.500131 23270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:05:11.501904 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 23202
I20260929 21:05:11.502094 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-1/wal/instance
I20260929 21:05:11.502528 388 external_mini_cluster.cc:1428] Setting key 4a50b6a0bf5c53177ad1eb480e9e79fc
I20260929 21:05:11.506510 23270 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46339 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 21:05:11 dist-test-slave-1dms krb5kdc[23115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260929 21:05:11.528241 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:11.509044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58864 (local address 127.0.97.62:46339)
0929 21:05:11.509225 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:11.509230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:11.510536 (+ 1306us) server_negotiation.cc:407] Connection header received
0929 21:05:11.512647 (+ 2111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:11.512655 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:11.512751 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:11.512893 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:11.514292 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:11.515429 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:11.516926 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:11.517712 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:11.517856 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:11.521414 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:05:11.521423 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:05:11.521431 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:05:11.521478 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:05:11.525168 (+ 3690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:11.525878 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:11.525886 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:11.525892 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:11.526048 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:11.526454 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:11.526459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:11.526462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:11.526871 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:05:11.527033 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:11.527648 (+ 615us) server_negotiation.cc:299] Negotiation successful
0929 21:05:11.527868 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 21:05:11.529311 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:11.509489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46339 (local address 127.0.0.1:58864)
0929 21:05:11.510339 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:11.510396 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 21:05:11.512370 (+ 1974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:05:11.513095 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:05:11.513104 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:05:11.513614 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:05:11.514112 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:05:11.514127 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:05:11.515578 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:05:11.515582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:05:11.516746 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:05:11.516758 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:05:11.517917 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:05:11.517921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:05:11.518106 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:11.518789 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:05:11.518814 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:05:11.521175 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:05:11.525314 (+ 4139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:05:11.525320 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:05:11.525333 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:05:11.525745 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:05:11.526172 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:05:11.526175 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:05:11.526177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:05:11.526342 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:05:11.527006 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:05:11.527011 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:05:11.527338 (+ 327us) client_negotiation.cc:769] Sending connection context
0929 21:05:11.527564 (+ 226us) client_negotiation.cc:240] Negotiation successful
0929 21:05:11.528060 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":323,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260929 21:05:11.531093 23270 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37807 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 21:05:11 dist-test-slave-1dms krb5kdc[23115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 21:05:11.547259 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:11.531904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37807 (local address 127.0.0.1:54570)
0929 21:05:11.532133 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:11.532156 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 21:05:11.532262 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:05:11.533221 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:05:11.533232 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:05:11.533722 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:05:11.533921 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:05:11.533935 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:05:11.535280 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:05:11.535288 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:05:11.536824 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:05:11.536860 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:05:11.537655 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:05:11.537663 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:05:11.537887 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:11.538501 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:05:11.538519 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:05:11.540612 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:05:11.544878 (+ 4266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:05:11.544886 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:05:11.544890 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:05:11.545261 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:05:11.545677 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:05:11.545680 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:05:11.545683 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:05:11.545814 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:05:11.546510 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:05:11.546513 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:05:11.546634 (+ 121us) client_negotiation.cc:769] Sending connection context
0929 21:05:11.546750 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 21:05:11.547011 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 21:05:11.547763 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:11.532108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54570 (local address 127.0.97.61:37807)
0929 21:05:11.532757 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:11.532762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:11.532787 (+ 25us) server_negotiation.cc:407] Connection header received
0929 21:05:11.532829 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:11.532894 (+ 65us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:11.533069 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:11.533184 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:11.534085 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:11.535123 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:11.536982 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:11.537505 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:11.537597 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:11.540830 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:05:11.540868 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:05:11.540886 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:05:11.540927 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:05:11.544687 (+ 3760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:11.545407 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:11.545419 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:11.545425 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:11.545569 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:11.545914 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:11.545921 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:11.545927 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:11.546382 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:05:11.546600 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:11.546941 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 21:05:11.547223 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":371,"threads_started":1}
I20260929 21:05:11.550115 23270 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32997 } has no permanent_uuid. Determining permanent_uuid...
I20260929 21:05:11.551235 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:11.550691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32997 (local address 127.0.0.1:46744)
0929 21:05:11.550891 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:11.551038 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 21:05:11.551496 23220 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32997: connect: Connection refused (error 111)
I20260929 21:05:11.553251 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:11.552736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32997 (local address 127.0.0.1:46748)
0929 21:05:11.552985 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:11.553095 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 21:05:11.553503 23270 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:32997: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32997: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260929 21:05:11.565140 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
/tmp/dist-test-taskdDfHCq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:32997
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46339,127.0.97.61:37807,127.0.97.60:32997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 21:05:11.602854 23270 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32997 } attempt: 1
I20260929 21:05:11.604038 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:11.603532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32997 (local address 127.0.0.1:46750)
0929 21:05:11.603744 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:11.603850 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 21:05:11.605072 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:11.604683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32997 (local address 127.0.0.1:46766)
0929 21:05:11.604823 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:11.604947 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 21:05:11.605268 23270 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:32997: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32997: connect: Connection refused (error 111)
I20260929 21:05:11.673485 23270 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32997 } attempt: 2
I20260929 21:05:11.674674 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:11.674140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32997 (local address 127.0.0.1:46780)
0929 21:05:11.674354 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:11.674486 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 21:05:11.675726 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:11.675312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32997 (local address 127.0.0.1:46794)
0929 21:05:11.675467 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:11.675558 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 21:05:11.675964 23270 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:32997: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32997: connect: Connection refused (error 111)
I20260929 21:05:11.728219 23195 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37807 } attempt: 5
Sep 29 21:05:11 dist-test-slave-1dms krb5kdc[23115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 21:05:11.742734 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:11.729043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37807 (local address 127.0.0.1:54584)
0929 21:05:11.729227 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:11.729248 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 21:05:11.729393 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:05:11.729816 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:05:11.729819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:05:11.730189 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:05:11.730357 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:05:11.730366 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:05:11.731838 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:05:11.731842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:05:11.732694 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:05:11.732705 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:05:11.733835 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:05:11.733840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:05:11.733970 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:11.734624 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:05:11.734642 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:05:11.736940 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:05:11.740558 (+ 3618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:05:11.740562 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:05:11.740565 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:05:11.741002 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:05:11.741371 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:05:11.741374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:05:11.741376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:05:11.741494 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:05:11.742066 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:05:11.742069 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:05:11.742193 (+ 124us) client_negotiation.cc:769] Sending connection context
0929 21:05:11.742294 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 21:05:11.742494 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 21:05:11.742972 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:11.729130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54584 (local address 127.0.97.61:37807)
0929 21:05:11.729319 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:11.729325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:11.729349 (+ 24us) server_negotiation.cc:407] Connection header received
0929 21:05:11.729530 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:11.729536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:11.729636 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:11.729753 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:11.730525 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:11.731647 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:11.732873 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:11.733664 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:11.733825 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:11.737123 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:05:11.737130 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:05:11.737135 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:05:11.737176 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:05:11.740428 (+ 3252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:11.741112 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:11.741117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:11.741121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:11.741263 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:11.741603 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:11.741608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:11.741611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:11.741955 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:05:11.742101 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:11.742445 (+ 344us) server_negotiation.cc:299] Negotiation successful
0929 21:05:11.742671 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 21:05:11.744064 23195 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32997 } has no permanent_uuid. Determining permanent_uuid...
I20260929 21:05:11.745155 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:11.744712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32997 (local address 127.0.0.1:46806)
0929 21:05:11.744904 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:11.744993 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 21:05:11.746157 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:11.745712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32997 (local address 127.0.0.1:46822)
0929 21:05:11.745909 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:11.746006 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 21:05:11.746403 23195 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:32997: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32997: connect: Connection refused (error 111)
I20260929 21:05:11.767201 23270 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32997 } attempt: 3
I20260929 21:05:11.768384 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:11.767829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32997 (local address 127.0.0.1:46826)
0929 21:05:11.768049 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:11.768165 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 21:05:11.769552 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:11.769084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32997 (local address 127.0.0.1:46834)
0929 21:05:11.769318 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:11.769413 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260929 21:05:11.769781 23270 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:32997: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32997: connect: Connection refused (error 111)
I20260929 21:05:11.797636 23195 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32997 } attempt: 1
I20260929 21:05:11.798898 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:11.798289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32997 (local address 127.0.0.1:46838)
0929 21:05:11.798546 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:11.798675 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 21:05:11.800045 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:11.799630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32997 (local address 127.0.0.1:46842)
0929 21:05:11.799799 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:11.799894 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 21:05:11.800313 23195 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:32997: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32997: connect: Connection refused (error 111)
I20260929 21:05:11.868543 23195 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32997 } attempt: 2
I20260929 21:05:11.869768 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:11.869210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32997 (local address 127.0.0.1:46844)
0929 21:05:11.869462 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:11.869570 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 21:05:11.870826 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:11.870446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32997 (local address 127.0.0.1:46860)
0929 21:05:11.870608 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:11.870690 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 21:05:11.871058 23195 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:32997: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32997: connect: Connection refused (error 111)
W20260929 21:05:11.892359 23277 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 21:05:11.892704 23277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 21:05:11.892839 23277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 21:05:11.893029 23277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 21:05:11.900089 23277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 21:05:11.900166 23277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 21:05:11.900221 23277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 21:05:11.900264 23277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 21:05:11.900317 23277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 21:05:11.909840 23277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46339,127.0.97.61:37807,127.0.97.60:32997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:32997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 21:05:11.911064 23277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 21:05:11.912300 23277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 21:05:11.912997 23270 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32997 } attempt: 4
I20260929 21:05:11.914155 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:11.913613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32997 (local address 127.0.0.1:46866)
0929 21:05:11.913865 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:11.913970 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 21:05:11.915189 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:11.914829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32997 (local address 127.0.0.1:46870)
0929 21:05:11.914981 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:11.915063 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 21:05:11.915417 23270 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:32997: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32997: connect: Connection refused (error 111)
W20260929 21:05:11.924662 23282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:05:11.924599 23285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 21:05:11.924638 23283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 21:05:11.925338 23277 server_base.cc:1034] running on GCE node
I20260929 21:05:11.926059 23277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 21:05:11.927353 23277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 21:05:11.928570 23277 hybrid_clock.cc:648] HybridClock initialized: now 1790715911928501 us; error 38 us; skew 500 ppm
I20260929 21:05:11.929008 23277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 21:05:11 dist-test-slave-1dms krb5kdc[23115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 21:05:11.931973 23277 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260929 21:05:11.933441 23277 webserver.cc:460] Webserver started at http://127.0.97.60:43043/ using document root <none> and password file <none>
I20260929 21:05:11.934072 23277 fs_manager.cc:359] Metadata directory not provided
I20260929 21:05:11.934154 23277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 21:05:11.934401 23277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 21:05:11.937213 23277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-2/data/instance:
uuid: "d6d4f71ad0fd4a5da2e4ffd5276802fd"
format_stamp: "Formatted at 2026-09-29 21:05:11 on dist-test-slave-1dms"
server_key: "e272dd8293cb4baecbec3e691c9ea19d"
server_key_iv: "45934a03e32a260e8518658021931948"
server_key_version: "encryptionkey@0"
I20260929 21:05:11.937911 23277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-2/wal/instance:
uuid: "d6d4f71ad0fd4a5da2e4ffd5276802fd"
format_stamp: "Formatted at 2026-09-29 21:05:11 on dist-test-slave-1dms"
server_key: "e272dd8293cb4baecbec3e691c9ea19d"
server_key_iv: "45934a03e32a260e8518658021931948"
server_key_version: "encryptionkey@0"
I20260929 21:05:11.942541 23277 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 21:05:11.945667 23292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 21:05:11.946925 23277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 21:05:11.947057 23277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-2/wal
uuid: "d6d4f71ad0fd4a5da2e4ffd5276802fd"
format_stamp: "Formatted at 2026-09-29 21:05:11 on dist-test-slave-1dms"
server_key: "e272dd8293cb4baecbec3e691c9ea19d"
server_key_iv: "45934a03e32a260e8518658021931948"
server_key_version: "encryptionkey@0"
I20260929 21:05:11.947218 23277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 21:05:11.963718 23277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 21:05:11.967852 23277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 21:05:11.968073 23277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 21:05:11.977264 23195 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32997 } attempt: 3
I20260929 21:05:11.978472 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:11.977984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32997 (local address 127.0.0.1:46878)
0929 21:05:11.978182 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:11.978273 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 21:05:11.979691 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:11.979198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32997 (local address 127.0.0.1:46892)
0929 21:05:11.979433 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:11.979531 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260929 21:05:11.979928 23195 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:32997: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32997: connect: Connection refused (error 111)
I20260929 21:05:11.986560 23277 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:32997
I20260929 21:05:11.986567 23343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:32997 every 8 connection(s)
I20260929 21:05:11.987854 23277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-2/data/info.pb
I20260929 21:05:11.988281 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu as pid 23277
I20260929 21:05:11.988462 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDfHCq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790715556572573-388-0/minicluster-data/master-2/wal/instance
I20260929 21:05:11.988976 388 external_mini_cluster.cc:1428] Setting key c858f7a8b9e16184e1c6144336b48bb7
I20260929 21:05:11.992465 23344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 21:05:12.001010 23344 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46339 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 21:05:12 dist-test-slave-1dms krb5kdc[23115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 21:05:12.012396 23347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:11.990813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46896 (local address 127.0.97.60:32997)
0929 21:05:11.991586 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:11.991598 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:11.991639 (+ 41us) server_negotiation.cc:407] Connection header received
0929 21:05:11.993440 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:11.993460 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:11.993835 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:11.994139 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:11.995860 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:11.997248 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:11.999008 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:11.999639 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:11.999763 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:12.004680 (+ 4917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:05:12.004689 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:05:12.004706 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:05:12.004757 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:05:12.008300 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:12.008969 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:12.008977 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:12.008985 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:12.009194 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:12.009565 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:12.009571 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:12.009575 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:12.010101 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:05:12.010406 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:12.010855 (+ 449us) server_negotiation.cc:299] Negotiation successful
0929 21:05:12.011253 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":347,"threads_started":1}
Sep 29 21:05:12 dist-test-slave-1dms krb5kdc[23115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 21:05:12.019909 23348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:12.004070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46339 (local address 127.0.0.1:58880)
0929 21:05:12.004784 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:12.004869 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0929 21:05:12.005057 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:05:12.005405 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:05:12.005413 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:05:12.006324 (+ 911us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:05:12.006750 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:05:12.006768 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.007745 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.007752 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:05:12.009288 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:05:12.009301 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.010011 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.010017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:05:12.010215 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:12.010847 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:05:12.010867 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:05:12.013352 (+ 2485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:05:12.017154 (+ 3802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:05:12.017167 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:05:12.017174 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:05:12.017711 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:05:12.018141 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:05:12.018144 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:05:12.018146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:05:12.018263 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:05:12.018843 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:05:12.018849 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:05:12.019004 (+ 155us) client_negotiation.cc:769] Sending connection context
0929 21:05:12.019200 (+ 196us) client_negotiation.cc:240] Negotiation successful
0929 21:05:12.019511 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":298,"threads_started":1}
I20260929 21:05:12.019981 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:12.004064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58880 (local address 127.0.97.62:46339)
0929 21:05:12.004254 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:12.004263 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:12.004989 (+ 726us) server_negotiation.cc:407] Connection header received
0929 21:05:12.005181 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:12.005185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:12.005250 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:12.005335 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:12.006912 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.007584 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.009412 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.009820 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.009971 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:12.013531 (+ 3560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:05:12.013536 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:05:12.013539 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:05:12.013573 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:05:12.017027 (+ 3454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:12.017836 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:12.017842 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:12.017846 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:12.018038 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:12.018386 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:12.018391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:12.018395 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:12.018723 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:05:12.018867 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:12.019303 (+ 436us) server_negotiation.cc:299] Negotiation successful
0929 21:05:12.019516 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 21:05:12.021652 23344 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37807 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 21:05:12 dist-test-slave-1dms krb5kdc[23115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 29 21:05:12 dist-test-slave-1dms krb5kdc[23115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 21:05:12.032591 23349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:12.015539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58886 (local address 127.0.97.62:46339)
0929 21:05:12.016113 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:12.016118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:12.016140 (+ 22us) server_negotiation.cc:407] Connection header received
0929 21:05:12.016198 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:12.016203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:12.016269 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:12.016374 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:12.017748 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.018926 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.020737 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.021476 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.021586 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:12.026420 (+ 4834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:05:12.026426 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:05:12.026431 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:05:12.026466 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:05:12.030090 (+ 3624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:12.030743 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:12.030749 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:12.030753 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:12.030894 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:12.031247 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:12.031252 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:12.031255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:12.031510 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:05:12.031772 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:12.032055 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 21:05:12.032267 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":256,"threads_started":1}
I20260929 21:05:12.036367 23348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:12.022677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37807 (local address 127.0.0.1:54594)
0929 21:05:12.022840 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:12.022857 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 21:05:12.022952 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:05:12.023290 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:05:12.023295 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:05:12.023699 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:05:12.023857 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:05:12.023867 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.025190 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.025195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:05:12.026093 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:05:12.026105 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.027072 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.027079 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:05:12.027263 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:12.027775 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:05:12.027797 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:05:12.030286 (+ 2489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:05:12.033919 (+ 3633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:05:12.033924 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:05:12.033928 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:05:12.034458 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:05:12.034825 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:05:12.034830 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:05:12.034833 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:05:12.034973 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:05:12.035478 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:05:12.035483 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:05:12.035637 (+ 154us) client_negotiation.cc:769] Sending connection context
0929 21:05:12.035774 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 21:05:12.036037 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 21:05:12.036486 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:12.022359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54594 (local address 127.0.97.61:37807)
0929 21:05:12.022530 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:12.022536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:12.022921 (+ 385us) server_negotiation.cc:407] Connection header received
0929 21:05:12.023064 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:12.023071 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:12.023156 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:12.023265 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:12.024012 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.025047 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.026231 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.026907 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.027046 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:12.030468 (+ 3422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:05:12.030473 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:05:12.030476 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:05:12.030505 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:05:12.033785 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:12.034554 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:12.034561 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:12.034565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:12.034697 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:12.035084 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:12.035089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:12.035092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:12.035354 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:05:12.035516 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:12.035876 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 21:05:12.036075 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 21:05:12.037405 23344 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32997 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 21:05:12 dist-test-slave-1dms krb5kdc[23115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 29 21:05:12 dist-test-slave-1dms krb5kdc[23115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 21:05:12.049521 23350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:12.034120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54604 (local address 127.0.97.61:37807)
0929 21:05:12.034649 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:12.034653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:12.034670 (+ 17us) server_negotiation.cc:407] Connection header received
0929 21:05:12.034707 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:12.034710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:12.034756 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:12.034867 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:12.036028 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.037135 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.038432 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.038947 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.039034 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:12.043647 (+ 4613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:05:12.043652 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:05:12.043655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:05:12.043682 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:05:12.046897 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:12.047517 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:12.047522 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:12.047526 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:12.047664 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:12.047993 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:12.048002 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:12.048006 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:12.048351 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:05:12.048601 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:12.048914 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 21:05:12.049115 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":240,"threads_started":1}
I20260929 21:05:12.050848 23348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:12.037971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32997 (local address 127.0.0.1:46906)
0929 21:05:12.038149 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:12.038167 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 21:05:12.038258 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:05:12.038568 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:05:12.038571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:05:12.038889 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:05:12.039050 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:05:12.039061 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.040143 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.040150 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:05:12.041655 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:05:12.041669 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.042401 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.042407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:05:12.042530 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:12.043046 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:05:12.043062 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:05:12.045025 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:05:12.048258 (+ 3233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:05:12.048261 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:05:12.048263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:05:12.048612 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:05:12.049030 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:05:12.049037 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:05:12.049042 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:05:12.049162 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:05:12.049812 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:05:12.049816 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:05:12.049971 (+ 155us) client_negotiation.cc:769] Sending connection context
0929 21:05:12.050115 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 21:05:12.050426 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":2}
I20260929 21:05:12.050894 23347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:12.038121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46906 (local address 127.0.97.60:32997)
0929 21:05:12.038254 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:12.038258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:12.038272 (+ 14us) server_negotiation.cc:407] Connection header received
0929 21:05:12.038356 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:12.038359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:12.038442 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:12.038536 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:12.039196 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.040013 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.041781 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.042238 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.042370 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:12.045192 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:05:12.045196 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:05:12.045199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:05:12.045225 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:05:12.048145 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:12.048743 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:12.048749 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:12.048753 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:12.048929 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:12.049272 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:12.049277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:12.049280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:12.049522 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:05:12.049687 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:12.050188 (+ 501us) server_negotiation.cc:299] Negotiation successful
0929 21:05:12.050578 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":159,"server-negotiator.queue_time_us":43}
I20260929 21:05:12.054322 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 21:05:12.056092 23344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6d4f71ad0fd4a5da2e4ffd5276802fd: Bootstrap starting.
I20260929 21:05:12.056567 23170 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c71f061c92ea49be9f1289e71ac10917"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58886
I20260929 21:05:12.059185 23344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6d4f71ad0fd4a5da2e4ffd5276802fd: Neither blocks nor log segments found. Creating new log.
I20260929 21:05:12.060211 23344 log.cc:824] T 00000000000000000000000000000000 P d6d4f71ad0fd4a5da2e4ffd5276802fd: Log is configured to *not* fsync() on all Append() calls
I20260929 21:05:12.062901 23344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6d4f71ad0fd4a5da2e4ffd5276802fd: No bootstrap required, opened a new log
I20260929 21:05:12.069911 23344 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6d4f71ad0fd4a5da2e4ffd5276802fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c71f061c92ea49be9f1289e71ac10917" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46339 } } peers { permanent_uuid: "5c122f501d7f4faca40a9808b28e7cd5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37807 } } peers { permanent_uuid: "d6d4f71ad0fd4a5da2e4ffd5276802fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32997 } }
I20260929 21:05:12.070226 23344 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6d4f71ad0fd4a5da2e4ffd5276802fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6d4f71ad0fd4a5da2e4ffd5276802fd, State: Initialized, Role: FOLLOWER
I20260929 21:05:12.070669 23344 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6d4f71ad0fd4a5da2e4ffd5276802fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c71f061c92ea49be9f1289e71ac10917" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46339 } } peers { permanent_uuid: "5c122f501d7f4faca40a9808b28e7cd5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37807 } } peers { permanent_uuid: "d6d4f71ad0fd4a5da2e4ffd5276802fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32997 } }
I20260929 21:05:12.071564 23352 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6d4f71ad0fd4a5da2e4ffd5276802fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c71f061c92ea49be9f1289e71ac10917" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46339 } } peers { permanent_uuid: "5c122f501d7f4faca40a9808b28e7cd5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37807 } } peers { permanent_uuid: "d6d4f71ad0fd4a5da2e4ffd5276802fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32997 } } }
I20260929 21:05:12.071950 23352 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6d4f71ad0fd4a5da2e4ffd5276802fd [sys.catalog]: This master's current role is: FOLLOWER
I20260929 21:05:12.072441 23344 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6d4f71ad0fd4a5da2e4ffd5276802fd [sys.catalog]: configured and running, proceeding with master startup.
W20260929 21:05:12.081595 23363 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d6d4f71ad0fd4a5da2e4ffd5276802fd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 21:05:12.081768 23363 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 21:05:12.079221 23277 master_runner.cc:186] Error getting master registration for 127.0.97.62:46339: OK, instance_id { permanent_uuid: "c71f061c92ea49be9f1289e71ac10917" instance_seqno: 1790715911014896 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 21:05:12.087494 23277 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 21:05:12.157157 23195 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32997 } attempt: 4
Sep 29 21:05:12 dist-test-slave-1dms krb5kdc[23115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 21:05:12.173583 23347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:12.158226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46912 (local address 127.0.97.60:32997)
0929 21:05:12.158483 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:12.158490 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:12.158548 (+ 58us) server_negotiation.cc:407] Connection header received
0929 21:05:12.158762 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:12.158768 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:12.158860 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:12.158972 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:12.159993 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.161209 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.162962 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.163771 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.163902 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:12.167860 (+ 3958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:05:12.167867 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:05:12.167872 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:05:12.167914 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:05:12.170764 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:12.171406 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:12.171412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:12.171416 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:12.171565 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:12.171978 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:12.171983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:12.171986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:12.172395 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:05:12.172576 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:12.172965 (+ 389us) server_negotiation.cc:299] Negotiation successful
0929 21:05:12.173171 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 21:05:12.173579 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:12.157994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32997 (local address 127.0.0.1:46912)
0929 21:05:12.158461 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:12.158482 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 21:05:12.158629 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:05:12.159025 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:05:12.159030 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:05:12.159606 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:05:12.159842 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:05:12.159855 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.161346 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.161352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:05:12.162808 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:05:12.162826 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.163960 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.163968 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:05:12.164194 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:12.164912 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:05:12.164937 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:05:12.167668 (+ 2731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:05:12.170909 (+ 3241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:05:12.170915 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:05:12.170919 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:05:12.171278 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:05:12.171682 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:05:12.171688 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:05:12.171691 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:05:12.171859 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:05:12.172519 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:05:12.172527 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:05:12.172691 (+ 164us) client_negotiation.cc:769] Sending connection context
0929 21:05:12.172875 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 21:05:12.173172 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325}
I20260929 21:05:12.179733 23195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c71f061c92ea49be9f1289e71ac10917: Bootstrap starting.
I20260929 21:05:12.183861 23195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c71f061c92ea49be9f1289e71ac10917: Neither blocks nor log segments found. Creating new log.
I20260929 21:05:12.185184 23195 log.cc:824] T 00000000000000000000000000000000 P c71f061c92ea49be9f1289e71ac10917: Log is configured to *not* fsync() on all Append() calls
I20260929 21:05:12.188562 23195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c71f061c92ea49be9f1289e71ac10917: No bootstrap required, opened a new log
I20260929 21:05:12.196136 23195 raft_consensus.cc:348] T 00000000000000000000000000000000 P c71f061c92ea49be9f1289e71ac10917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c71f061c92ea49be9f1289e71ac10917" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46339 } } peers { permanent_uuid: "5c122f501d7f4faca40a9808b28e7cd5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37807 } } peers { permanent_uuid: "d6d4f71ad0fd4a5da2e4ffd5276802fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32997 } }
I20260929 21:05:12.196555 23195 raft_consensus.cc:729] T 00000000000000000000000000000000 P c71f061c92ea49be9f1289e71ac10917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c71f061c92ea49be9f1289e71ac10917, State: Initialized, Role: FOLLOWER
I20260929 21:05:12.197083 23195 consensus_queue.cc:260] T 00000000000000000000000000000000 P c71f061c92ea49be9f1289e71ac10917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c71f061c92ea49be9f1289e71ac10917" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46339 } } peers { permanent_uuid: "5c122f501d7f4faca40a9808b28e7cd5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37807 } } peers { permanent_uuid: "d6d4f71ad0fd4a5da2e4ffd5276802fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32997 } }
I20260929 21:05:12.198071 23366 sys_catalog.cc:455] T 00000000000000000000000000000000 P c71f061c92ea49be9f1289e71ac10917 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c71f061c92ea49be9f1289e71ac10917" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46339 } } peers { permanent_uuid: "5c122f501d7f4faca40a9808b28e7cd5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37807 } } peers { permanent_uuid: "d6d4f71ad0fd4a5da2e4ffd5276802fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32997 } } }
I20260929 21:05:12.198634 23366 sys_catalog.cc:458] T 00000000000000000000000000000000 P c71f061c92ea49be9f1289e71ac10917 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 21:05:12.199474 23195 sys_catalog.cc:564] T 00000000000000000000000000000000 P c71f061c92ea49be9f1289e71ac10917 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:05:12.199541 23170 raft_consensus.cc:482] T 00000000000000000000000000000000 P c71f061c92ea49be9f1289e71ac10917 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 21:05:12.199781 23170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c71f061c92ea49be9f1289e71ac10917 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:05:12.202890 23170 raft_consensus.cc:504] T 00000000000000000000000000000000 P c71f061c92ea49be9f1289e71ac10917 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c71f061c92ea49be9f1289e71ac10917" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46339 } } peers { permanent_uuid: "5c122f501d7f4faca40a9808b28e7cd5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37807 } } peers { permanent_uuid: "d6d4f71ad0fd4a5da2e4ffd5276802fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32997 } }
I20260929 21:05:12.204334 23170 leader_election.cc:288] T 00000000000000000000000000000000 P c71f061c92ea49be9f1289e71ac10917 [CANDIDATE]: Term 1 election: Requested vote from peers 5c122f501d7f4faca40a9808b28e7cd5 (127.0.97.61:37807), d6d4f71ad0fd4a5da2e4ffd5276802fd (127.0.97.60:32997)
I20260929 21:05:12.205312 23319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c71f061c92ea49be9f1289e71ac10917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d6d4f71ad0fd4a5da2e4ffd5276802fd"
I20260929 21:05:12.205897 23319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6d4f71ad0fd4a5da2e4ffd5276802fd [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:05:12.209702 23319 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d6d4f71ad0fd4a5da2e4ffd5276802fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c71f061c92ea49be9f1289e71ac10917 in term 1.
I20260929 21:05:12.210758 23145 leader_election.cc:302] T 00000000000000000000000000000000 P c71f061c92ea49be9f1289e71ac10917 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c71f061c92ea49be9f1289e71ac10917, d6d4f71ad0fd4a5da2e4ffd5276802fd; no voters:
I20260929 21:05:12.211256 23366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c71f061c92ea49be9f1289e71ac10917 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 21:05:12.214633 23270 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32997 } attempt: 5
W20260929 21:05:12.220798 23298 tablet.cc:2369] T 00000000000000000000000000000000 P d6d4f71ad0fd4a5da2e4ffd5276802fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 21:05:12.225338 23366 raft_consensus.cc:686] T 00000000000000000000000000000000 P c71f061c92ea49be9f1289e71ac10917 [term 1 LEADER]: Becoming Leader. State: Replica: c71f061c92ea49be9f1289e71ac10917, State: Running, Role: LEADER
I20260929 21:05:12.226011 23366 consensus_queue.cc:237] T 00000000000000000000000000000000 P c71f061c92ea49be9f1289e71ac10917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c71f061c92ea49be9f1289e71ac10917" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46339 } } peers { permanent_uuid: "5c122f501d7f4faca40a9808b28e7cd5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37807 } } peers { permanent_uuid: "d6d4f71ad0fd4a5da2e4ffd5276802fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32997 } }
Sep 29 21:05:12 dist-test-slave-1dms krb5kdc[23115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
W20260929 21:05:12.254263 23149 tablet.cc:2369] T 00000000000000000000000000000000 P c71f061c92ea49be9f1289e71ac10917: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 21:05:12.255570 23347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:12.216354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46914 (local address 127.0.97.60:32997)
0929 21:05:12.216597 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:12.216606 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:12.216631 (+ 25us) server_negotiation.cc:407] Connection header received
0929 21:05:12.216720 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:12.216728 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:12.216867 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:12.216992 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:12.218586 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.220324 (+ 1738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.226119 (+ 5795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.226915 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.227017 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:12.236779 (+ 9762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:05:12.236790 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:05:12.236799 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:05:12.236888 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:05:12.244923 (+ 8035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:12.247084 (+ 2161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:12.247131 (+ 47us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:12.247138 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:12.247539 (+ 401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:12.248617 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:12.248625 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:12.248667 (+ 42us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:12.249669 (+ 1002us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:05:12.250610 (+ 941us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:12.251630 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0929 21:05:12.252818 (+ 1188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 21:05:12.256295 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:12.215393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32997 (local address 127.0.0.1:46914)
0929 21:05:12.215906 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0929 21:05:12.215934 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 21:05:12.216088 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 21:05:12.217256 (+ 1168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 21:05:12.217265 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 21:05:12.218018 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0929 21:05:12.218309 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:05:12.218327 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.223439 (+ 5112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.223450 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:05:12.225829 (+ 2379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 21:05:12.225871 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.227744 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.227750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 21:05:12.228463 (+ 713us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:12.229899 (+ 1436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 21:05:12.229965 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0929 21:05:12.236338 (+ 6373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 21:05:12.245287 (+ 8949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:05:12.245298 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:05:12.245305 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:05:12.246821 (+ 1516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:05:12.247905 (+ 1084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 21:05:12.247914 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 21:05:12.247948 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0929 21:05:12.248325 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 21:05:12.250028 (+ 1703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 21:05:12.250038 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 21:05:12.250831 (+ 793us) client_negotiation.cc:769] Sending connection context
0929 21:05:12.251970 (+ 1139us) client_negotiation.cc:240] Negotiation successful
0929 21:05:12.253600 (+ 1630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327}
I20260929 21:05:12.264303 23373 sys_catalog.cc:455] T 00000000000000000000000000000000 P c71f061c92ea49be9f1289e71ac10917 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c71f061c92ea49be9f1289e71ac10917. Latest consensus state: current_term: 1 leader_uuid: "c71f061c92ea49be9f1289e71ac10917" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c71f061c92ea49be9f1289e71ac10917" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46339 } } peers { permanent_uuid: "5c122f501d7f4faca40a9808b28e7cd5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37807 } } peers { permanent_uuid: "d6d4f71ad0fd4a5da2e4ffd5276802fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32997 } } }
I20260929 21:05:12.264791 23373 sys_catalog.cc:458] T 00000000000000000000000000000000 P c71f061c92ea49be9f1289e71ac10917 [sys.catalog]: This master's current role is: LEADER
I20260929 21:05:12.265785 23387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 21:05:12.266227 23350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:12.235331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54610 (local address 127.0.97.61:37807)
0929 21:05:12.235548 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:12.235557 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:12.242488 (+ 6931us) server_negotiation.cc:407] Connection header received
0929 21:05:12.242932 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:12.242938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:12.243027 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:12.243094 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:12.245645 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.247100 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.250927 (+ 3827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.251819 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.251942 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:12.258100 (+ 6158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:05:12.258111 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:05:12.258119 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:05:12.258175 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:05:12.262999 (+ 4824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:12.263767 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:12.263773 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:12.263776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:12.263932 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:12.264332 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:12.264339 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:12.264343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:12.264759 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:05:12.265032 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:12.265631 (+ 599us) server_negotiation.cc:299] Negotiation successful
0929 21:05:12.265902 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 21:05:12.271327 23387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 21:05:12.271613 23270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c122f501d7f4faca40a9808b28e7cd5: Bootstrap starting.
I20260929 21:05:12.276588 23270 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c122f501d7f4faca40a9808b28e7cd5: Neither blocks nor log segments found. Creating new log.
I20260929 21:05:12.287012 23319 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d6d4f71ad0fd4a5da2e4ffd5276802fd [term 1 FOLLOWER]: Refusing update from remote peer c71f061c92ea49be9f1289e71ac10917: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:05:12.288121 23373 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c71f061c92ea49be9f1289e71ac10917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6d4f71ad0fd4a5da2e4ffd5276802fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:05:12.289171 23270 log.cc:824] T 00000000000000000000000000000000 P 5c122f501d7f4faca40a9808b28e7cd5: Log is configured to *not* fsync() on all Append() calls
I20260929 21:05:12.319962 23270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c122f501d7f4faca40a9808b28e7cd5: No bootstrap required, opened a new log
I20260929 21:05:12.315188 23128 master_runner.cc:186] Error getting master registration for 127.0.97.61:37807: OK, instance_id { permanent_uuid: "5c122f501d7f4faca40a9808b28e7cd5" instance_seqno: 1790715911490326 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 21:05:12.329530 23128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 21:05:12.338838 23347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:12.248877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46916 (local address 127.0.97.60:32997)
0929 21:05:12.257644 (+ 8767us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:12.257654 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:12.261676 (+ 4022us) server_negotiation.cc:407] Connection header received
0929 21:05:12.261912 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:12.261918 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:12.262079 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:12.262281 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:12.289936 (+ 27655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.291354 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.300936 (+ 9582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.302060 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.302191 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:12.331138 (+ 28947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:05:12.331145 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:05:12.331151 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:05:12.331203 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:05:12.335022 (+ 3819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:12.336135 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:12.336141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:12.336144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:12.336317 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:12.336790 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:12.336796 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:12.336799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:12.337206 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:05:12.337459 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:12.338085 (+ 626us) server_negotiation.cc:299] Negotiation successful
0929 21:05:12.338384 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8258,"thread_start_us":13702,"threads_started":1}
I20260929 21:05:12.339505 23349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:12.234683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58898 (local address 127.0.97.62:46339)
0929 21:05:12.235150 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:12.235158 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:12.248345 (+ 13187us) server_negotiation.cc:407] Connection header received
0929 21:05:12.285016 (+ 36671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:12.285026 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:12.285140 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:12.285231 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:12.293717 (+ 8486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.295094 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.306473 (+ 11379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.310247 (+ 3774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.310428 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:12.326071 (+ 15643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 21:05:12.326078 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 21:05:12.326085 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 21:05:12.326132 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 21:05:12.330062 (+ 3930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:12.331073 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:12.331078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:12.331081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:12.331405 (+ 324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 21:05:12.336147 (+ 4742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 21:05:12.336152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 21:05:12.336156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 21:05:12.337030 (+ 874us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 21:05:12.337529 (+ 499us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:12.338386 (+ 857us) server_negotiation.cc:299] Negotiation successful
0929 21:05:12.338863 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329}
I20260929 21:05:12.347544 23352 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6d4f71ad0fd4a5da2e4ffd5276802fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c71f061c92ea49be9f1289e71ac10917. Latest consensus state: current_term: 1 leader_uuid: "c71f061c92ea49be9f1289e71ac10917" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c71f061c92ea49be9f1289e71ac10917" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46339 } } peers { permanent_uuid: "5c122f501d7f4faca40a9808b28e7cd5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37807 } } peers { permanent_uuid: "d6d4f71ad0fd4a5da2e4ffd5276802fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32997 } } }
I20260929 21:05:12.347880 23352 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6d4f71ad0fd4a5da2e4ffd5276802fd [sys.catalog]: This master's current role is: FOLLOWER
I20260929 21:05:12.351532 23366 sys_catalog.cc:455] T 00000000000000000000000000000000 P c71f061c92ea49be9f1289e71ac10917 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c71f061c92ea49be9f1289e71ac10917" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c71f061c92ea49be9f1289e71ac10917" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46339 } } peers { permanent_uuid: "5c122f501d7f4faca40a9808b28e7cd5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37807 } } peers { permanent_uuid: "d6d4f71ad0fd4a5da2e4ffd5276802fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32997 } } }
I20260929 21:05:12.351859 23366 sys_catalog.cc:458] T 00000000000000000000000000000000 P c71f061c92ea49be9f1289e71ac10917 [sys.catalog]: This master's current role is: LEADER
I20260929 21:05:12.356793 23270 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c122f501d7f4faca40a9808b28e7cd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c71f061c92ea49be9f1289e71ac10917" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46339 } } peers { permanent_uuid: "5c122f501d7f4faca40a9808b28e7cd5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37807 } } peers { permanent_uuid: "d6d4f71ad0fd4a5da2e4ffd5276802fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32997 } }
I20260929 21:05:12.357403 23387 catalog_manager.cc:1347] Generated new cluster ID: 1afdbee5a4594f4187ca527b9cf026c7
I20260929 21:05:12.357496 23387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 21:05:12.357542 23270 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c122f501d7f4faca40a9808b28e7cd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c122f501d7f4faca40a9808b28e7cd5, State: Initialized, Role: FOLLOWER
I20260929 21:05:12.358430 23270 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c122f501d7f4faca40a9808b28e7cd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c71f061c92ea49be9f1289e71ac10917" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46339 } } peers { permanent_uuid: "5c122f501d7f4faca40a9808b28e7cd5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37807 } } peers { permanent_uuid: "d6d4f71ad0fd4a5da2e4ffd5276802fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32997 } }
I20260929 21:05:12.370857 23352 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6d4f71ad0fd4a5da2e4ffd5276802fd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c71f061c92ea49be9f1289e71ac10917" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c71f061c92ea49be9f1289e71ac10917" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46339 } } peers { permanent_uuid: "5c122f501d7f4faca40a9808b28e7cd5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37807 } } peers { permanent_uuid: "d6d4f71ad0fd4a5da2e4ffd5276802fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32997 } } }
I20260929 21:05:12.371116 23352 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6d4f71ad0fd4a5da2e4ffd5276802fd [sys.catalog]: This master's current role is: FOLLOWER
I20260929 21:05:12.373471 23396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c122f501d7f4faca40a9808b28e7cd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c71f061c92ea49be9f1289e71ac10917" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46339 } } peers { permanent_uuid: "5c122f501d7f4faca40a9808b28e7cd5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37807 } } peers { permanent_uuid: "d6d4f71ad0fd4a5da2e4ffd5276802fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32997 } } }
I20260929 21:05:12.373961 23396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c122f501d7f4faca40a9808b28e7cd5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 21:05:12.376808 23245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c71f061c92ea49be9f1289e71ac10917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5c122f501d7f4faca40a9808b28e7cd5"
I20260929 21:05:12.378741 23244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c122f501d7f4faca40a9808b28e7cd5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 21:05:12.382395 23244 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5c122f501d7f4faca40a9808b28e7cd5 [term 1 FOLLOWER]: Refusing update from remote peer c71f061c92ea49be9f1289e71ac10917: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 21:05:12.385632 23245 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5c122f501d7f4faca40a9808b28e7cd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c71f061c92ea49be9f1289e71ac10917 in term 1.
I20260929 21:05:12.387689 23366 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c71f061c92ea49be9f1289e71ac10917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c122f501d7f4faca40a9808b28e7cd5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 21:05:12.392377 23270 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c122f501d7f4faca40a9808b28e7cd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 21:05:12.407254 23387 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 21:05:12.409165 23387 catalog_manager.cc:1504] Loading token signing keys...
I20260929 21:05:12.432865 23401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c122f501d7f4faca40a9808b28e7cd5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c71f061c92ea49be9f1289e71ac10917" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c71f061c92ea49be9f1289e71ac10917" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46339 } } peers { permanent_uuid: "5c122f501d7f4faca40a9808b28e7cd5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37807 } } peers { permanent_uuid: "d6d4f71ad0fd4a5da2e4ffd5276802fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32997 } } }
I20260929 21:05:12.433279 23401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c122f501d7f4faca40a9808b28e7cd5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 21:05:12.433400 23373 sys_catalog.cc:455] T 00000000000000000000000000000000 P c71f061c92ea49be9f1289e71ac10917 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c71f061c92ea49be9f1289e71ac10917" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c71f061c92ea49be9f1289e71ac10917" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46339 } } peers { permanent_uuid: "5c122f501d7f4faca40a9808b28e7cd5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37807 } } peers { permanent_uuid: "d6d4f71ad0fd4a5da2e4ffd5276802fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32997 } } }
I20260929 21:05:12.433693 23373 sys_catalog.cc:458] T 00000000000000000000000000000000 P c71f061c92ea49be9f1289e71ac10917 [sys.catalog]: This master's current role is: LEADER
I20260929 21:05:12.433610 23396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c122f501d7f4faca40a9808b28e7cd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c71f061c92ea49be9f1289e71ac10917. Latest consensus state: current_term: 1 leader_uuid: "c71f061c92ea49be9f1289e71ac10917" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c71f061c92ea49be9f1289e71ac10917" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46339 } } peers { permanent_uuid: "5c122f501d7f4faca40a9808b28e7cd5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37807 } } peers { permanent_uuid: "d6d4f71ad0fd4a5da2e4ffd5276802fd" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32997 } } }
I20260929 21:05:12.433849 23396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c122f501d7f4faca40a9808b28e7cd5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 21:05:12.445520 23387 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c71f061c92ea49be9f1289e71ac10917: Generated new TSK 0
I20260929 21:05:12.447635 23387 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 21:05:12.469281 23411 catalog_manager.cc:1259] Loaded cluster ID: 1afdbee5a4594f4187ca527b9cf026c7
I20260929 21:05:12.469390 23411 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5c122f501d7f4faca40a9808b28e7cd5: loading cluster ID for follower catalog manager: success
I20260929 21:05:12.473970 23411 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5c122f501d7f4faca40a9808b28e7cd5: acquiring CA information for follower catalog manager: success
I20260929 21:05:12.476645 23411 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5c122f501d7f4faca40a9808b28e7cd5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 21:05:12.568051 388 mini_kdc.cc:348] Time spent kdestroy: real 0.102s user 0.004s sys 0.017s
W20260929 21:05:12.621613 23349 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:05:12.600811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58908 (local address 127.0.97.62:46339)
0929 21:05:12.601060 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:12.601066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:12.617471 (+ 16405us) server_negotiation.cc:407] Connection header received
0929 21:05:12.619925 (+ 2454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:12.619932 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:12.620021 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:12.620097 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:12.621318 (+ 1221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58908: BlockingRecv error: recv got EOF from 127.0.0.1:58908 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
W20260929 21:05:12.626463 23350 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:05:12.602635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54622 (local address 127.0.97.61:37807)
0929 21:05:12.602812 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:12.602818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:12.623728 (+ 20910us) server_negotiation.cc:407] Connection header received
0929 21:05:12.624579 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:12.624585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:12.624667 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:12.624739 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:12.626188 (+ 1449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54622: BlockingRecv error: recv got EOF from 127.0.0.1:54622 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260929 21:05:12.627874 23347 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:05:12.618180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46922 (local address 127.0.97.60:32997)
0929 21:05:12.618353 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:12.618358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:12.624204 (+ 5846us) server_negotiation.cc:407] Connection header received
0929 21:05:12.624970 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:12.624976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:12.625060 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:12.625127 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:12.627611 (+ 2484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46922: BlockingRecv error: recv got EOF from 127.0.0.1:46922 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 21:05:12.673123 23350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:12.656245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54636 (local address 127.0.97.61:37807)
0929 21:05:12.656437 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:12.656443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:12.665415 (+ 8972us) server_negotiation.cc:407] Connection header received
0929 21:05:12.665758 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:12.665766 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:12.666012 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:12.666153 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:05:12.666838 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.667995 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.670067 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.670739 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.670834 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:12.671540 (+ 706us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:05:12.672268 (+ 728us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:05:12.672384 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:12.672667 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 21:05:12.672756 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 21:05:12.708510 23347 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:05:12.706158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46930 (local address 127.0.97.60:32997)
0929 21:05:12.706359 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:12.706365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:12.706943 (+ 578us) server_negotiation.cc:407] Connection header received
0929 21:05:12.707164 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:12.707170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:12.707298 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:12.707417 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:12.708263 (+ 846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46930: BlockingRecv error: recv got EOF from 127.0.0.1:46930 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 21:05:12.722045 23350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:12.714791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54646 (local address 127.0.97.61:37807)
0929 21:05:12.714973 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:12.714980 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:12.715453 (+ 473us) server_negotiation.cc:407] Connection header received
0929 21:05:12.715735 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:12.715741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:12.715829 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:12.715893 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:05:12.716515 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.717689 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.719703 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.720469 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.720640 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:12.721238 (+ 598us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:05:12.721382 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:05:12.721494 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:12.721657 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 21:05:12.721739 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 21:05:12.734845 23347 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:05:12.729737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46934 (local address 127.0.97.60:32997)
0929 21:05:12.729920 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:12.729926 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:12.733349 (+ 3423us) server_negotiation.cc:407] Connection header received
0929 21:05:12.733525 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:12.733531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:12.733621 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:12.733734 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:12.734576 (+ 842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46934: BlockingRecv error: recv got EOF from 127.0.0.1:46934 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 21:05:12.745990 23350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:12.739524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54652 (local address 127.0.97.61:37807)
0929 21:05:12.739709 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:12.739715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:12.740180 (+ 465us) server_negotiation.cc:407] Connection header received
0929 21:05:12.740326 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:12.740332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:12.740411 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:12.740508 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:05:12.740955 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.742107 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.744022 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.744566 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.744688 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:12.745187 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:05:12.745320 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:05:12.745413 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:12.745599 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 21:05:12.745683 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 21:05:12.760738 23347 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:05:12.758340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46938 (local address 127.0.97.60:32997)
0929 21:05:12.758547 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:12.758553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:12.759098 (+ 545us) server_negotiation.cc:407] Connection header received
0929 21:05:12.759323 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:12.759329 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:12.759429 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:12.759542 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:12.760494 (+ 952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46938: BlockingRecv error: recv got EOF from 127.0.0.1:46938 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 21:05:12.782989 23350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:12.775725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54666 (local address 127.0.97.61:37807)
0929 21:05:12.775971 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:12.775978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:12.776526 (+ 548us) server_negotiation.cc:407] Connection header received
0929 21:05:12.776748 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:12.776755 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:12.776901 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:12.777014 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:05:12.777523 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.778791 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.780748 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.781473 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.781614 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:12.782080 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:05:12.782253 (+ 173us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:05:12.782355 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:12.782565 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 21:05:12.782655 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
W20260929 21:05:12.799067 23347 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:05:12.794538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46948 (local address 127.0.97.60:32997)
0929 21:05:12.794812 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:12.794818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:12.797533 (+ 2715us) server_negotiation.cc:407] Connection header received
0929 21:05:12.797715 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:12.797721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:12.797826 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:12.797950 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:12.798842 (+ 892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46948: BlockingRecv error: recv got EOF from 127.0.0.1:46948 (error 108)
Metrics: {"server-negotiator.queue_time_us":148}
I20260929 21:05:12.822072 23350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:12.814913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54674 (local address 127.0.97.61:37807)
0929 21:05:12.815100 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:12.815110 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:12.815652 (+ 542us) server_negotiation.cc:407] Connection header received
0929 21:05:12.815863 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:12.815874 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:12.815991 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:12.816101 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:05:12.816643 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.817794 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.819777 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.820499 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.820632 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:12.821151 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:05:12.821293 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:05:12.821399 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:12.821610 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 21:05:12.821707 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260929 21:05:12.832008 23347 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:05:12.829845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46964 (local address 127.0.97.60:32997)
0929 21:05:12.830035 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:12.830041 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:12.830506 (+ 465us) server_negotiation.cc:407] Connection header received
0929 21:05:12.830691 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:12.830697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:12.830798 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:12.830912 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:12.831767 (+ 855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46964: BlockingRecv error: recv got EOF from 127.0.0.1:46964 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 21:05:12.859383 23350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:12.852301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54680 (local address 127.0.97.61:37807)
0929 21:05:12.852516 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:12.852522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:12.853054 (+ 532us) server_negotiation.cc:407] Connection header received
0929 21:05:12.853270 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:12.853278 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:12.853401 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:12.853507 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:05:12.853985 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.855211 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.857235 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.857931 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.858081 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:12.858562 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:05:12.858700 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:05:12.858805 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:12.858965 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 21:05:12.859056 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260929 21:05:12.869059 23347 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:05:12.866845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46968 (local address 127.0.97.60:32997)
0929 21:05:12.867039 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:12.867044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:12.867585 (+ 541us) server_negotiation.cc:407] Connection header received
0929 21:05:12.867762 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:12.867768 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:12.867860 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:12.867959 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:12.868817 (+ 858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46968: BlockingRecv error: recv got EOF from 127.0.0.1:46968 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 21:05:12.910780 23350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:12.904619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54684 (local address 127.0.97.61:37807)
0929 21:05:12.904888 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:12.904895 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:12.905277 (+ 382us) server_negotiation.cc:407] Connection header received
0929 21:05:12.905505 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:12.905510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:12.905578 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:12.905658 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:05:12.906179 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.906977 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.909010 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.909456 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.909553 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:12.910031 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:05:12.910145 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:05:12.910219 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:12.910385 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 21:05:12.910455 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260929 21:05:12.918753 23347 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:05:12.916891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46982 (local address 127.0.97.60:32997)
0929 21:05:12.917090 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:12.917097 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:12.917282 (+ 185us) server_negotiation.cc:407] Connection header received
0929 21:05:12.917535 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:12.917539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:12.917622 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:12.917703 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:12.918491 (+ 788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46982: BlockingRecv error: recv got EOF from 127.0.0.1:46982 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 21:05:12.991760 23350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:12.986271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54688 (local address 127.0.97.61:37807)
0929 21:05:12.986487 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:12.986493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:12.986869 (+ 376us) server_negotiation.cc:407] Connection header received
0929 21:05:12.987019 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:12.987025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:12.987122 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:12.987229 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:05:12.987620 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.988627 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.989969 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:12.990604 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:12.990734 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:12.991054 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:05:12.991189 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:05:12.991273 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:12.991396 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 21:05:12.991474 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260929 21:05:13.000435 23347 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 21:05:12.998122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46992 (local address 127.0.97.60:32997)
0929 21:05:12.998420 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:12.998430 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:12.999058 (+ 628us) server_negotiation.cc:407] Connection header received
0929 21:05:12.999267 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:12.999270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:12.999340 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:12.999424 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 21:05:13.000186 (+ 762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46992: BlockingRecv error: recv got EOF from 127.0.0.1:46992 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 21:05:13.083750 23363 catalog_manager.cc:1259] Loaded cluster ID: 1afdbee5a4594f4187ca527b9cf026c7
I20260929 21:05:13.083882 23363 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d6d4f71ad0fd4a5da2e4ffd5276802fd: loading cluster ID for follower catalog manager: success
I20260929 21:05:13.086313 23363 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d6d4f71ad0fd4a5da2e4ffd5276802fd: acquiring CA information for follower catalog manager: success
I20260929 21:05:13.088054 23363 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d6d4f71ad0fd4a5da2e4ffd5276802fd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 21:05:13.136682 23350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:13.131888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54700 (local address 127.0.97.61:37807)
0929 21:05:13.132064 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:13.132069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:13.132426 (+ 357us) server_negotiation.cc:407] Connection header received
0929 21:05:13.132583 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:13.132587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:13.132655 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:13.132744 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:05:13.133163 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:13.133865 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:13.135187 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:13.135597 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:13.135739 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:13.136041 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:05:13.136143 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:05:13.136210 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:13.136367 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 21:05:13.136428 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 21:05:13.146903 23347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:13.141803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47002 (local address 127.0.97.60:32997)
0929 21:05:13.141970 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:13.141974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:13.142288 (+ 314us) server_negotiation.cc:407] Connection header received
0929 21:05:13.142411 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:13.142415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:13.142559 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:13.142654 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:05:13.143008 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:13.143659 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:13.144988 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:13.145379 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:13.145470 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:13.145891 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:05:13.146416 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:05:13.146523 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:13.146600 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 21:05:13.146662 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 21:05:13.159715 23350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:13.152262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54714 (local address 127.0.97.61:37807)
0929 21:05:13.152444 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:13.152450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:13.153005 (+ 555us) server_negotiation.cc:407] Connection header received
0929 21:05:13.153255 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:13.153260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:13.153335 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:13.153399 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:05:13.154494 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:13.155510 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:13.157836 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:13.158408 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:13.158479 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:13.158925 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:05:13.159045 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:05:13.159126 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:13.159324 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 21:05:13.159404 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 21:05:13.160310 23518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:13.152483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58910 (local address 127.0.97.62:46339)
0929 21:05:13.152987 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:13.152994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:13.153013 (+ 19us) server_negotiation.cc:407] Connection header received
0929 21:05:13.153125 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:13.153131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:13.153365 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:13.153533 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:05:13.154335 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:13.155461 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:13.157267 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:13.157675 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:13.157815 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:13.159027 (+ 1212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:05:13.159544 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:05:13.159639 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:13.159884 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 21:05:13.159962 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":643,"threads_started":1}
I20260929 21:05:13.160364 23347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 21:05:13.153133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47014 (local address 127.0.97.60:32997)
0929 21:05:13.153342 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 21:05:13.153350 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 21:05:13.154387 (+ 1037us) server_negotiation.cc:407] Connection header received
0929 21:05:13.154585 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 21:05:13.154589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 21:05:13.154648 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 21:05:13.154751 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 21:05:13.155179 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:13.156046 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:13.158227 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 21:05:13.158930 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 21:05:13.159093 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 21:05:13.159574 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 21:05:13.159684 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 21:05:13.159754 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 21:05:13.159987 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 21:05:13.160097 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 21:05:13.164947 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 23128
I20260929 21:05:13.165926 23190 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:05:13.277329 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 23128
W20260929 21:05:13.298079 23219 connection.cc:535] server connection from 127.0.0.1:54584 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 21:05:13.298233 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 23202
I20260929 21:05:13.299335 23265 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:05:13.406710 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 23202
I20260929 21:05:13.427021 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu pid 23277
I20260929 21:05:13.428177 23339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 21:05:13.534919 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDfHCq/build/asan/bin/kudu with pid 23277
2026-09-29T21:05:13Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4293 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9155 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (356928 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST